• Harry Newton's picture

    PyCon 2019 Cardiff

    Harry Newton / September 19, 2019
  • I attended PyCon in Cardiff last week, on the 13th and 14th of September. This is a selection of my notes during my time there, highlighting the talks and discussions I found useful. My full notes are available should anyone want to read through them, including talks that I haven't listed here.
     

    AstroPi – Python on the ISS (Ben Nuttall)

    There are two Raspberry Pis on the ISS. One has a camera pointed into the space station, one has an IR camera pointed at earth. Each can be logged, take photos and have real time processing. They are each equipped with the 'Sense Hat' for Raspberry Pi, an attachment that has a lot of different sensors on it including an 8x8 LED display for simple LED text displays. The two Raspberry Pis were sent to space in December 2015 with Tim Peake.

    The ESA (European Space agency) run educational programmes to get kids interested in space. The kids write code that runs on them. There is a competition to write programmes with these RPi units that Tim Peake can use. Open to ESA member states (most of the EU and Canada). The two missions are Mission Zero which gets 30 seconds of runtime on the Pi, the aim of which is to write a 'Hello world' programme. You have access to sensors but not the camera. Submission is through the web emulator on trinket.io. The second is Mission SpaceLab which gets 3 hours of runtime on the Pi. This runs a science experiment. There are two themes – Life on Earth and Life in space. A more complex experiment with more phases.

    One project from last year captured, by chance, an approaching SpaceX shuttle making its approach to the ISS. It was briefly designated as a UFO… until it was identified.

    Crew time is requested for access to RPis via an SD card in a laptop. Also via SSH access now that they have LAN access.

    Any photos take inside the ISS cannot be kept for privacy reasons, and any submissions also get given a map of where the ISS was over the 30secs/3hours of programme runtime.

    Space Lab programme examples:

    • Detect crew presence near RPi
    • Log data to see what happens
    • Environmental conditions
    • Time-lapse
    • Global issues
    • Wildfires
    • Deforestation
    • Greenery
    • Shrinkage of lakes

    Libraries

    • Picamera
    • Pyephem – location of ISS
    • GDAL – Geospacial data
    • Numpy, scipi, pandras – science and maths libraries
    • Tensorflow, opencv, and more

     

    This was an example of how Python is already being used in space and how it can be accessed from earth.

     

    Asynchronous web development with Flask (Miguel Grinberg)

    'Processes' and 'Threads' are also for this purpose. Concurrency in Python.

    Asyncio is an async library for Python based on coroutines. See also: Twisted, Gevent, Eventlet.

    Async is highly scalable with long lived http requests. It is great for platforms with no support for Processes or Threads. Most processing time is from the rest of the application, so while asyncio seems so much faster, it actually only saves a small amount of time overall in a full application.

    The Flask framework cannot work with coroutines, therefore Asyncio will not work here. However, it will work with Greenlets. Quart and Sanic are good alternatives to Flask for asynchio compatibility.

     

    Writing micro-services in Python…Sure! But which framework? (Emmanuelle Delescolle)

    Their opinion is that Django is ‘too bloated’ for micro-services. This is a list of what they recommend on different frameworks:

    What we as developers want:

    • Good docs
    • Good ecosystem
    • Easy switch between environments
    • Shell access for debugging
    • Auto documentation of endpoints
    • Browsable API
    • Common project architecture

    What we want as sysadmin:

    • Good documentation
    • Common tooling/software
    • Managed processes
    • Logging
    • Easy switch between environment’s
    • Hot reload

    What we want as product owners:

    • Widespread framework
    • Good documentation
    • Common project architecture
    • No bikeshedding (Spending disproportionate amount of time on something less important)
    • Some performance
    • Document APIs
    • Testing

    FLASK

    • ‘Free-form’
    • No boilerplate
    • Wide and somewhat scattered ecosystem
    • Preferred SQLAlchemy – Oauth2
    • (Apparently Flask stated as a joke)

     

    Docs/Ecosystem

    • Django wins
    • Flask is headache
    • Pyramid is neutral
    • FastAPI is scarce

     

    Shell access

    • OK across all
    • Missing in FlaskAPI

     

    Docs

    • Out of the box in Django and FlaskAPI
    • API NOT browsable in FastAPI yet
    • Easy in Pyramid
    • Fastidious in Flask

     

    Testing framework

    • All testable

     

    Common project architecture

    • Django - not much choice
    • Bit more in Pyramid
    • Flask and FastAPI let you do whatever you want

     

    Common Tooling

    • Django/Flask/Pyramid are WSGI apps
    • uWSGI provides process management and hot reload
    • FastAPI is ASGI

     

    Logging

    • Standard Python logging

     

    Easy switch between environments

    • Pyramid wins
    • Flask has something
    • Django has something else
    • FastAPI…not yet

     

    Bikeshedding

    (The analogy of spending too much time on something not as important (building the bike shed) as the rest of the product (building the house).

    • There will always be some

     

    Performance

    • FastAPI uses less CPU
    • Flask uses less RAM

     

    Conclusions

    Use whatever you are comfortable with. All have pros and cons.

    A sensible conclusion, but definitely useful to have a list of pros and cons for when deciding between frameworks on these metrics in the future.

     

    How to write readable tests (David Seddon)

    Tests tend to be less readable than production code. Readable tests clarify scope and are living documentation. They also facilitate change.

    Six principles of writing good tests:

    1. Profit from the work of others
      • Tools already out there that are useful e.g. PyTest, WebTest, Factory Boy
    2. Put naming to work
      • Test functions named as what they do/test they perform
      • Variables should be named as what they represent
    3. Show only what matters
      • Private methods/functions to hide unimportant stuff
      • By the same hand, don’t hide too much!
    4. Show your working.
      • Use a simplified data model. Option is to transform the actual result into a simplified result that can be compared to expected results.
    5. Don’t repeat yourself
      • Don’t repeat yourself
      • Single source of truth (Applies to production code)
    6. Maintainability and readability (Applied to all code, including tests)
      • Sub-classing tests
      • Arrange, Act, Assert (GIVEN, WHEN, THEN)

     

    Aim High

    Tests tend to be treated as second class citizens. Do with tests what's already done with production code, such as peer reviewing tests. Always review tests first

    This is a good takeaway for us at Edina, as including these principles in writing our tests will no doubt ensure their quality. Adding peer reviews to our test writing would also be a good start in this.

     

    Keynote - Leadership and identify in the pan-African Python movement (Marlene Mhangami)

    An interesting talk from an organiser of PyCon Africa. There were 26 countries represented in PyCon Africa, Ghana. 323 attendees, a big mix of nationalities and cultures

    Speaking of Python meet-ups, there was one Python user group meetup in Antarctica.

     

    I spoke to a representative from Anvil

    After getting a demo and speaking to a representative from Anvil, these are my main points about it:

    • The entire environment is setup in Python, all in Anvil.
    • There is no payment required, you can always try out the free version.
    • If we want to use AWS without migrating the entire project, AWS can be linked to.
    • Pip install can work as normal

    Anvil may be of interest to us if wanting to setup a web application quickly and not worry about including all the other components such as CSS, which framework to use, etc., as it is all contained.

     

    Robotic Microscopy for everyone (Joel Collins)

    This talk was focused on open source hardware e.g. Arduino, RepRap 3D printer (You can reproduce the 3d printer, using the 3d printer)

    Using the main component of a microscope, the majority of the rest is mechanics. The main component can be added into a 3D printed ‘mechanics’ part for a fraction of the cost. For example, flexible plastic squares are used to approximate the accuracy of a slider. Name ‘Flexures’.

    These cheap microscopes are being using with real data form real patients in Tanzania.

    Imaging software running on the Pi, detects this microscope is running nearby, and runs an autofocus on the microscope. Pan around the image with arrow keys to move the microscope.

     

    Rapid prototyping scalable Python services using AWS Chalice (Mashhood Rastgar)

    Being an AWS talk, this might be of interest to the ICED project. Chalice is a python framework to deploy python on AWS lambda.

    The documentation is 1 page, no need to look into Lambda and read all the information and reams of documentation there. You can also use Chalice CLI to run locally, connecting to your AWS account. Chalice can automatically convert a returned Python dictionary to JSON format and send it as a correct response.

    Using Web-sockets for streaming data. Web-sockets themselves are deployed on API Gateway. Whenever there is a message, API Gateway triggers the appropriate gateway.

     

    Two very useful commands:

    • Chalice deploy – deploys everything. Outputs the endpoint for the rest API.
    • Chalice delete – deletes everything deployed

     

    However, It still has issues:

    • the function cold starts (500ms first time)
    • API gateway idle timeout
    • Lambda execution limits
    • Lambda is more expensive
    • Package size limitations (Max 250 MB)Libraries

     

    From the audience's questions:

    Zappa is another option. A community driven solution.

    Using Terraform and chalice together - Separate deployment pipelines for both. It’s simpler and there shouldn’t be issues. CloudFormation should be similar, but this was not answered.

     

    Young Coders show and tell

    There was a youbng coders day, for primary and young seconday school sctudents. They had their own stream of wqorkshops to teach them programming on Raspberry Pis. At the end of teh day, they had a show and tell.

    The ‘Box of justice’ – judging dead cats. Random cat picture, random cat name, good/bad judgement. Rather impressive both with the programme and presentation.

    Bob the dragon – light up nose. A crowd favourite.