Commit graph

190 commits

Author SHA1 Message Date
Romain Bazile f801d74c0e imager: bugfix for wrong message received
If we stop the imager when the pump is still running, we were removing the callback treating the messages from the pump.
The effect is that if a status message has already been sent by the pump, the imager would parse the message with its normal code. So yeah, not nice.
I've now integrated all message treatment in one function, and pump_callback disappeared!
2020-12-15 16:44:01 +01:00
Romain Bazile 87fb866942 integrity: typo in error message 2020-12-15 16:08:15 +01:00
Romain Bazile f55182d79c raspimjpeg: remove unnecessary scripts 2020-12-08 17:31:23 +01:00
Romain Bazile fde4b166d4 segmenter: remove description from metadata 2020-12-08 16:52:50 +01:00
Romain Bazile dfdcf66be5 raspimjpeg: replace FIFO location 2020-12-08 16:52:22 +01:00
Romain Bazile 9852c3c469 imager: remove spaces in path 2020-12-08 16:51:55 +01:00
Romain Bazile 5312f37906 gallery: add all local files 2020-12-05 13:28:33 +01:00
Romain Bazile ede52d59db double all i2c calls and change nginx config in update script 2020-12-05 11:39:49 +01:00
Romain Bazile 6ce27f177d hardware.json: add wb gain settings 2020-12-05 03:22:47 +01:00
Romain Bazile 849d6188fd rename nginx template file 2020-12-05 03:13:01 +01:00
Romain Bazile 69b8782fb4 flow: icon of monitoring tab and update nginx config 2020-12-05 03:04:46 +01:00
Romain Bazile 7cc9aa4845 stepper: solve bug if flowrate is 0 2020-12-05 02:59:52 +01:00
Romain Bazile 1faf91ff51 Remove thumbsup and replace by simple nginx autoindex 2020-12-05 02:22:20 +01:00
Romain Bazile 93a7b5658d update: another typo, time to stop and go home 2020-12-03 18:07:50 +01:00
Romain Bazile bb8b3225d9 update: fix major fuckup 2020-12-03 18:00:40 +01:00
Romain Bazile ae13f07f45 fix typos 2020-12-03 17:28:54 +01:00
Romain Bazile b3d417e3b9 imager bug if object_date is not set 2020-12-03 16:45:01 +01:00
Romain Bazile d9aab5fc0b fix imager bug 2020-12-03 16:29:12 +01:00
Romain Bazile a03f3fb6aa imager: run thumbsup after every picture taken 2020-12-03 14:30:59 +01:00
Romain Bazile 841d9b8831 gallery: finishing installing thumbsup 2020-12-03 14:19:13 +01:00
Romain Bazile bbd3146f7c update: finable around npm crappy ownership issues 2020-12-03 13:59:23 +01:00
Romain Bazile aec7669ac6 update: add thumbsup install 2020-12-03 13:18:19 +01:00
Romain Bazile e2d6a21d0f thumbsup: add config 2020-12-03 13:13:09 +01:00
Romain Bazile a72b819b5a Flow: addition of log output, refactoring of GPS coordinates handling, refactoring of datetime management 2020-12-02 19:26:27 +01:00
Romain Bazile 13307476e2 imager add uuid for each acquisition and check for id reuse 2020-12-02 19:25:59 +01:00
Romain Bazile 8e70723310 add usb backup UI 2020-12-01 21:12:29 +01:00
Romain Bazile 054244f32c display: center text 2020-11-30 12:05:35 +01:00
Romain Bazile 9a0fef4411 first display use release, show machine name 2020-11-30 11:52:27 +01:00
Romain Bazile cbed5e4d18 main: add ready light 2020-11-30 11:45:59 +01:00
Romain Bazile ff59c0cac6 update the updater 2020-11-30 02:46:40 +01:00
Romain Bazile 51ff3b7375 update the light library
also replaces smbus by smbus2 to allow for cleaner setups
2020-11-30 00:17:30 +01:00
Romain Bazile 156286a8da add sample_gear_net_opening and sample_concentrated_sample_volume
improve #44
2020-11-29 23:48:54 +01:00
Romain Bazile 63102172fc add sleep_before, closes #48 2020-11-29 14:28:31 +01:00
Romain Bazile f04c2bcab6 add larger gain control range (up to 8.0) 2020-11-28 22:25:59 +01:00
Romain Bazile dcec3d01e4 retrieve camera name 2020-11-27 18:57:43 +01:00
Romain Bazile 594eeb5690 add white balance switch 2020-11-27 18:39:09 +01:00
Romain Bazile ebdf7876ce update: final update to flow 2020-11-27 12:53:54 +01:00
Romain Bazile fd741afc5a update: add git stash drop 2020-11-27 12:45:49 +01:00
Romain Bazile ac2d44a24e addition of white balance gain slider 2020-11-27 12:45:22 +01:00
Romain Bazile dd66f1adf0 update: improvement to the script 2020-11-27 11:58:01 +01:00
Romain Bazile f1b49e672b main.json: update to update code ;-) 2020-11-27 11:53:21 +01:00
Romain Bazile 4f5c59c061 imager: remove minimum volume check (done in Node-Red now) 2020-11-27 11:31:18 +01:00
Romain Bazile bcb1e245f0 usb_backup: fix typo and add purge of files 2020-11-27 11:30:49 +01:00
Romain Bazile a62891eb6a light: catch exceptions 2020-11-27 11:30:29 +01:00
Romain Bazile 1cf3d1df12 Fix bugs all around 2020-11-25 17:01:14 +01:00
Romain Bazile 05dc317095 imager: add support for white balance 2020-11-24 17:31:11 +01:00
Romain Bazile 3bb27b9e15 integrity: this and udev rule closes #15 2020-11-20 18:03:55 +01:00
Romain Bazile 6009e07cd6 first release of the integrity check mechanism 2020-11-20 18:03:55 +01:00
Romain Bazile ae29f8597e Main: print the machine name 2020-11-16 17:53:23 +01:00
Romain Bazile f27c9823a1 Integration of raspimjpeg
The source code of the modified version of raspimjpeg used by the PlanktoScope is hosted [here](https://github.com/PlanktonPlanet/userland/tree/master/host_applications/linux/apps/raspicam).

We moved away from picamera to improve the stability of the system. This commit should closes the elusive #51 and #54.

After a lot of tests, it was clear that even a forked version of picamera was not going to solve our problem while also improving on our capturing speed.

The modified version of RaspiMJPEG does that.
RaspiMJPEG streams the image flux to /dev/shm/mjpeg/cam.jpg . This is passively reloaded by the streaming server (still written in python).

RaspiMJPEG is controlled through a special file called a Pipe which is used to send it commands.
The list is available [here](https://github.com/PlanktonPlanet/userland/tree/master/host_applications/linux/apps/raspicam/Config.md).

The raspimjpeg binary is distributed directly in this repository, in the folder `scripts/raspimjpeg/bin/`.
2020-11-16 17:53:23 +01:00
Romain Bazile fcb57ed075 segmenter: import libraries when needed to make startup faster 2020-11-16 17:53:23 +01:00
Romain Bazile 98c51828e0 stepper: reduce method size 2020-11-16 17:53:23 +01:00
Romain Bazile 12712c241e imager: reduce methods size 2020-11-16 17:53:23 +01:00
Romain Bazile 4bcc9f1153 switch off bandit error that are not pertinent 2020-11-14 09:45:14 +01:00
Romain Bazile 736a09ff5f imager: catch BrokenPipeError 2020-11-14 09:45:14 +01:00
Romain Bazile 83f45d6c2f add microstep configuration option 2020-11-14 09:45:14 +01:00
Romain Bazile b66776a5fb sourcery.ai cleanup 2020-11-14 09:45:14 +01:00
Romain Bazile a5a76979b9 addition of uuidname module 2020-11-14 09:45:14 +01:00
Romain Bazile 1530c71fb0 Python: add/remove todos 2020-10-21 21:25:59 +02:00
Romain Bazile 06d9facb3d segmenter: fix morphocut error with objects files in img folder 2020-10-21 21:25:59 +02:00
Romain Bazile 3ce7f6aa5e stepper: fix pump bug where it doesn't stop 2020-10-21 21:25:59 +02:00
Romain Bazile 2cf0ff8341 main.py: change default log rotation/retention settings 2020-10-21 21:25:59 +02:00
Romain Bazile e7ea63a43a Scripts README 2020-10-07 12:25:50 +02:00
Romain Bazile 140a3cf0de License! Martyr proposal 2020-10-07 12:25:50 +02:00
Romain Bazile e89cbdc2aa python: reminder to add tests 2020-10-07 12:22:52 +02:00
Romain Bazile 1a42f05941 python: improve log messages 2020-10-07 12:22:52 +02:00
Romain Bazile 2c9a6df20f segmenter: try to add support for printing object_id 2020-10-07 12:22:52 +02:00
Romain Bazile 20e07a110d stepper: add support for waveshare hat, for #11 2020-10-07 12:22:52 +02:00
Romain Bazile 1488459954 Change default img and export directories, closes #55 2020-10-07 12:22:52 +02:00
Romain Bazile a35b70e0fb segmenter: bugfixes 2020-10-07 12:22:52 +02:00
Romain Bazile fbacf21016 segmenter: typo fix in mqtt topic 2020-09-28 21:40:13 +02:00
Romain Bazile 06ef401a3b
Extraction and refactor of the python code from node-red flow
The rationale for this rewrite is to improve the readability, the modularity, the reliability and the future-proofing of the main python script.

All in all, this is now 124 commits that are going to be squashed and merged together, spanning more than two weeks of development and testing.

Please test away this release and break things. An upgrading guide will be published in the coming days, along with a new image for people to use if they don't want to upgrade on their own.

Read along if you want to know all the goodies!

As a starter, the python script was extracted from the main flow, and now lives in its own files at `PlantonScope/scripts/*`.

We set up the auto formatting of the code by using [Black](https://github.com/psf/black). This make the code clearer and uniform. We are using the default settings, so if you just install Black and set your editor to format on save using it, you should be good to go.

The code is separated in four main processes, each with a specific set of responsibilities:
- The main process controls all the others, starts everything up and cleans up on shutdown
- The stepper process manages the stepper movements. It's now possible to have simultaneous movements of both motors (this closes #38 ).
- The imager process controls the camera and the streaming server via a state machine.
- The segmenter process manages the segmentation and its outputs. The segmentation happens recursively in all folders in `/home/pi/PlanktonScope/img/`. Each folder has its own output archive, and bug #26 is now closed.


Those processes communicates together using MQTT and json messages. Each message is adressed to one topic. The high level topic controls which process receives the message. The details of each topic is at the end of this commit message.

Every imaging sessions has now its own folder, under the `img` root. Metadata are saved individually for every session, in a JSON file in the same directory as the pictures.

The configuration is not parsed from `config.json` anymore and passed directly through MQTT messages to the concerned process.

A new configuration file has been created: `hardware.json`. This file contains information related to your specific hardware configuration. You can choose to reverse the connection of the motors for example, but you can also define specific speed limits and steps number for your pump and focus stage. This will make it easier for people who wants to experiment with different kind of hardware. It's not necessary to have this file though. If it doesn't exists, the default configuration will be applied.

The code is architectured around 6 modules and about 10 classes. I encourage you to have a look at the files, they're pretty straightforward to understand.

There is a lot of work left around the node-red code refactoring, dashboard ui improvements, better and clearer LED messages, OLED screen integration and finer control of the segmentation process, but this is quite good for now.


Here is the topic lists for MQTT and the corresponding messages.
- actuator :  This topic adresses the stepper control thread
              No publication under this topic should happen from the python process
  - actuator/pump :   Control of the pump
                      The message is a json object
                      {"action":"move", "direction":"FORWARD", "volume":10, "flowrate":1}
                      to move 10mL forward at 1mL/min
                      action can be "move" or "stop"
                      Receive only
  - actuator/focus :  Control of the focus stage
                      The message is a json object, speed is optional
                      {"action":"move", "direction":"UP", "distance":0.26, "speed":1}
                      to move up 10mm
                      action can be "move" or "stop"
                      Receive only
- imager/image :      This topic adresses the imaging thread
                      Is a json object with
                      {"action":"image","sleep":5,"volume":1,"nb_frame":200}
                      sleep in seconds, volume in mL
                      Can also receive a config update message:
                      {"action":"config","config":[...]}
                      Can also receive a camera settings message:
                      {"action":"settings","iso":100,"shutter_speed":40}
                      Receive only
- segmenter/segment : This topic adresses the segmenter process
                      Is a json object with
                      {"action":"segment"}
                      Receive only
- status :    This topics sends feedback to Node-Red
              No publication or receive at this level
  - status/pump :     State of the pump
                      Is a json object with
                      {"status":"Start", "time_left":25}
                      Status is one of Started, Ready, Done, Interrupted
                      Publish only
  - status/focus :    State of the focus stage
                      Is a json object with
                      {"status":"Start", "time_left":25}
                      Status is one of Started, Ready, Done, Interrupted
                      Publish only
  - status/imager :   State of the imager
                      Is a json object with
                      {"status":"Start", "time_left":25}
                      Status is one of Started, Ready, Completed or 12_11_15_0.1.jpg has been imaged.
                      Publish only
  - status/segmenter :   Status of the segmentation
      - status/segmenter/name
      - status/segmenter/object_id
      - status/segmenter/metric




Here is the original commit history:
* Extract python main.py from flow

* Fix bug in server addresses

These addresses should be the loopback device instead of the network 
address of the device. Using the loopback address will not necessitate 
to update the script when the network address changes.

* clean up picamera import

* changes to main python and flow:
update MQTT requests address to localhost (bugfix)
update streaming output address to nothing
update main flow to remove python script references and location

* Automatically initialise imaging led on startup to off state.

* Add the ability to invert outputs of the motor

We added a key to config.json "hardware_config" with a subkey 
"stepper_reverse". If this key is present in the config file and set to 
1, the output of the motors are inversed (stepper2 becomes the pump 
motor and stepper1 the focus motor)

* move all non main script to a subfolder

* add __init__.py to package

* light module rewrite

* json cleanup and absolute path for config file

* light.py forgot to import subprocess

#oups

* Add command to turn the leds off

* Auto formatting of main.py

I've used Black with default settings, see https://github.com/psf/black

* First commit of stepper.py
Pump parameters still needs to be checked and tuned.

* addition of hardware details in config.json

* Introduce hardware.json to replace the `hardware_config` of config.json

* stepper.py: calibration, typos

* creates the MQTT_Client class

* pump_max_speed is now in ml/min to help readability

* forgot to add self to the class def

* addition of threading capabilities to stepper.py (UNTESTED)

* mqtt: fix topic bug

* remove counter

* mqtt add doc about topics

* stepper.py creates an "actuator/*/state" topic

* stepper.py: rename mqtt_client to pump_client

* mqtt.py: add details about topics

* stepper.py: rename pump_client to actuator_client

* topic was not split properly and a part was lost

* switch to f-strings for mqtt.py

* cosmetic update

* stepper.py: folder name will be planktoscope change calls

* hardware.json became more straightforward

* stepper.py syntax bugs

* stepper.py addition of a received stop command

* stepper.py: update to max travel distance

* stepper.py: several typos here

* rename folder

* main.py: reword to reflect folder rename

* main.py: remove logic that has been moved to stepper.py and mqtt.py

* main.py: update to add mqtt imaging client

* mqtt.py: make command and args local to class and output more verbose

* make stepper.py a class

* main.py: instantiate stepper class and call it

* main.py: name mqtt client

* update to main.json to reflect main.py changes

* fix bugs around pump control

* update flows to latest version from Thibault

* distance can be a small value, and definitevely should be a float.

* unify mqtt topics

* unify mqtt output in the main flow

* first logger implementation, uses loguru

* mqtt: add reason to on_connect

* mqtt: add on_disconnect handler

* stepper: add more logger calls for debug mainly

* main: add levels for logger

* imager.py: first move of the imager logic

* imager: time import cleanup

* imager: morphocut import cleanup

* imager: skimage import cleanup

* imager: finishing import cleanup

* imager: Class creation - WIP

Also provides a fix for #26 (see line 190).

* imager: threading is needed for Condition()

* streamer: get the streamer server its own file

* imager: creates start_camera and get the server creation out

* imager: subclass multiprocessing.Process

* imager: get Pipeline creation its own function

* imager: cleanup of self calls

* main: code removal and corresponding calls to newly created classes

* imager: various formatting changes

* main: management of signal shutdown

* add requirements.txt

* mqtt: messages are now json objects

Also, addition of a flag on receiving a new message

* mqtt: make message private and add logic to synchronise

* stepper: creates the stepper class

* stepper: use the new class

* stepper: uses the new logic

* stepper: add the shutdown event

* stepper: add shutdown method

* main: add shutdown event

* imager: graceful shutdown

* stepper: nicer way of checking the Eevnt

* self is a required first argument for a method in a class

Especially if you use said class private members!

* python: various typos and small errors in import

* stepper: create mqtt client during init

* stepper: instanciate the mqtt client inside run

Otherwise it's not accessible from inside the loop. It's a PITA,
more information at  https://stackoverflow.com/questions/17172878/using-pythons-multiprocessing-process-class

* stepper: little bugs and typos all around

* mqtt: add shutdown method

* mqtt: add connect in init

* stepper: fix bugs, sanitize inputs

* stepper: work on delay prediction improvements

* stepper: json is mean, double quote are mandatory inside

* mqtt: add details about message exchanged

* imager: first implementation of json messages

* main.json: add new tab for RPi management + json for payloads

* imager: add state_machine class

* stepper: publish last will

* imager: major refactor

* main: make streaming server process a daemon

* mqtt: insert debug statement on close

* main: reorder imports

* imager: make it work!

Reinsert the streaming server logic in there, because there is a problem with the synchronisation part otherwise.
Also, eventually, StreamingOuput() will have to be made not global

Final very critical learning: it's super duper important to make sure the memory split is at least 256Meg for the GPU.
Chaos ensues otherwise

* main: changes to accomodate the streamer/imager fusino

* imager_state_machine: insert states transition description

* stepper: cleanup of code

* segmenter: creation of the class

* python: include segmenter changes

* remove unused files

* stepper: check existence of hardware.json

* main.json: changes to reflect the python script evolution

* remove unecessary TODOs and add some others

* main: add check for config and directories

* imager: update_config is implemented and we have better management of directories now

* segmenter: now work recursively in all folders

* flow: the configuration is now sent via mqtt

* segmenter: better manage pipeline error

* segmenter: declaration of archive_fn in init

* imager: small bugs and typos

* main: add uniqueID output

* imager: add the camera settings message

We can now update the ISO, shutter speed and resolution from Node-Red

* package.json: update dependencies
2020-09-28 11:05:27 +02:00
tpollina 5007ba35b5
Delete mqtt_pump_focus_image_segment_strem.py 2020-07-25 11:20:33 +02:00
tpollina 961146adbc
Delete mqtt_pump_focus_image.py 2020-07-25 11:20:25 +02:00
tpollina c474b3a7e6
Delete Morphocut_segmentation.py 2020-07-14 16:56:17 +02:00
tpollina 889d2eedae
Delete StackFlow_v3.py 2020-07-14 16:56:04 +02:00
thibaut pollina a80fae1af5 update README.md 2020-02-09 00:20:14 -08:00
thibaut pollina 60bdc3ede6 Clean up useless files 2020-02-09 00:15:36 -08:00
tpollina ff12a2e4e4
Create mqtt_pump_focus_image_segment_strem.py 2020-02-04 21:00:12 -08:00
tpollina ba0cf44e74
Update mqtt_nodered.json 2020-02-04 20:59:16 -08:00
tpollina 29183d9fb1
Update mqtt_nodered.json 2020-02-04 08:08:30 +01:00
tpollina 86e783ceee
Create mqtt_pump_focus_image_v2.py 2020-02-04 08:07:58 +01:00
tpollina 9f1609ded4
Update mqtt_pump_focus_image.py 2020-02-02 04:20:55 +01:00
tpollina 78de7302ad
Update mqtt_pump_focus_image.py 2020-02-01 21:25:49 +01:00
tpollina af90680da7
Update mqtt_pump_focus_image.py 2020-01-31 08:49:32 +01:00
tpollina b0353fc63b
Create mqtt_nodered.json 2020-01-30 19:44:27 -08:00
tpollina 58fa23b0eb
Create mqtt_pump_focus_image.py 2020-01-30 19:42:39 -08:00
tpollina cfeccf3faa
Create Morphocut_segmentation.py 2020-01-30 17:53:37 +01:00
tpollina b67f62746f
Create StackFlow_v3.py 2020-01-30 17:52:00 +01:00
tpollina 945d3a65fc
Rename scripts/flows.json to flows/flows.json 2020-01-27 16:29:58 -08:00
tpollina dca9e1f600
Rename scripts/flows_planktonscope.json to flows/flows_planktonscope.json 2020-01-27 16:29:38 -08:00
tpollina 3c3b281be2
Rename scripts/flow_villefranche.json to flows/flow_villefranche.json 2020-01-27 16:29:17 -08:00
tpollina 934c5cfeb8
Add files via upload 2020-01-27 14:40:43 -08:00
tpollina f9927cd90d
Create flow_villefranche.json 2020-01-27 19:10:43 +01:00
tpollina b45e0babfe
Update focus.py 2020-01-15 15:46:19 +01:00
tpollina 22045c9e8e
Delete recover.py 2020-01-15 15:45:11 +01:00
tpollina d129734d62
Add files via upload 2020-01-08 14:32:46 +01:00
tpollina 1f46ea7e07
Update focus.py 2019-12-19 04:57:32 -08:00
tpollina 3e796d83bb
Rename scripts/install.rst to docs/install.rst 2019-12-16 19:10:02 -08:00
tpollina 469a7bca03
Rename install.rst to scripts/install.rst 2019-12-16 19:09:49 -08:00