Following up on yesterday’s noodling about getting voice > voice using ChatGPT working, I thought I’d try hooking it up to a real life object. In this case a Cosmic Unicorn from Pimoroni.

Take a look:

This is super easy using Python. It’s unreal!

I’m using a system message for ChatGPT to set the behaviour for the assistant:

{"role": "system", "content": """
Convert this text to a programmatic command, and add a short confirmation message:
Example: Make the LEDs red
Output: {'target':'LEDs','result':{'r':255,'g':0,'b':0},'message':'OK, I made the LEDs red'}
"""
}

Should be pretty easy to adapt this for more controls..

Let me know if you’d like me to share the code!

Though to make this slightly different, this uses Wake Word and Voice Activity Detection using PicoVoice, so it’s kinda like an Alexa.. ish..?

The flow is: App starts > wait for the wake word (Helios — you can train your own wake words) > start recording the mic, and use Cobra VAD to detect when the user stops speaking, or wait a max of 6 seconds > save audio to file > send to Whisper API > get text back > send to ChatGPT API > get the response back > send it to Google Cloud Text To Speech API (en-GB-Wavenet-B) > get the MP3 back > play it via mpg321.

Only a few hours playing about with the APIs, and really fun!

Edit: I’ve now hooked this up to a physical device

This is mainly for Google, as I couldn’t find anything that actually did this in OSM. I’d have thought there would be an option to draw the borders of a state/country etc including the coastline and land borders, but apparently not.

Here’s the code for Overpass Turbo:

rel["ISO3166-2"="GB-SCT"]["type"="boundary"]["boundary"="administrative"]["admin_level"="4"];
(
  way(r)["maritime" != "yes"];
  area["ISO3166-2"="GB-SCT"][admin_level=4];
  way(area)["natural"="coastline"];
);

out geom;

You can see it here: https://overpass-turbo.eu/s/1rGa

I think this will work with any of the ISO 3166–2 codes, eg: GB-SCT, GB-ENG, IT-78, etc — as long as OSM supports them.

Let me know if you found it useful!

Let me know if I’m missing any!

https://www.toymakr3d.com

https://fab365.net

https://www.gambody.com/

https://www.patreon.com/mysterymakers

https://hex3dpatreon.com/

https://www.patreon.com/mrbaddeley

https://nikkoindustriesmembership.com/

https://www.myminifactory.com/

https://cults3d.com/en

https://sanix3d.com/

https://www.printables.com/

https://www.thingiverse.com/

https://www.do3d.com

https://www.patreon.com/FlexiFactory

Let’s go!

Andy Touch on Twitter: "Id like to see your prove that theory! I disagree but willing to change my mind if you can convince the AI to make a Flappy Bird clone. / Twitter"

Id like to see your prove that theory! I disagree but willing to change my mind if you can convince the AI to make a Flappy Bird clone.

“in python, make a flappy birds clone”

It generated the following Python:

Ok but did it run? It did! Repeatedly hit the spacebar to move up, gravity pulls you down.

There’s no obstacles though. Let’s add some:

Does it work? Yup!

I want more obstacles though, so let’s just ask ChatGPT for that:

Here it is in action:

Let’s add some colour:

It works:

Finally, lets add collision detection and a game over screen:

The final result:

I’m impressed! This only took a few minutes in total, and I can’t wait to see what future generations of ChatGPT is capable of.

While ChatGPT can work with many languages, I’m particularly excited about Python, due to the amount of free packages available (over 400k on PyPl). I have another blog post on ChatGPT here where I have some experiments with face recognition.

If you’re interested in the source code for this Flappy Birds “clone”, it can be found here.

Original thread here: https://twitter.com/yezzer/status/1599336356357894144

The future is here. I’ve seen it. I’ve used it. It has blown my mind.

ChatGPT has been released in free research preview and it is incredible.

While ChatGPT can do many different things, it’s the programming side that I’m particularly interested in.

You can ask it — in plain english — to generate code for you by describing what you want the code to do. Recently I’ve been using MicroPython and Python a lot, so focussed on that.

Here’s some examples:

Prompt:
“in python, open an image then draw a red alpha’d filed rectangle around any human faces. Show the rectangle properties aligned under the rectangle. Make the text red and bold”

Execution: (on AI faces)

Prompt:
“in python, open a video, then draw a red alpha’d filed rectangle around any human faces in each frame. Show the rectangle properties aligned under the rectangle. Make the text red and bold.”

Execution:

More in that series here: https://twitter.com/yezzer/status/1599031876416520192

Prompt:
“in python, write a web service that will return a live feed of my webcam”

Execution:

Prompt: “in python, open an image, and place a dog loaded from https://dog.ceo/api/breeds/image/random/ on top of the faces”
+ “each face should show a different dog”
+ “the dogs should be RGB not greyscale”
+ asking it to fix some errors (it fixes it’s own code too!)

Execution:

Prompt:
“write me python to open an image and draw a grid on top. the grid spacing should be 12 pixels”

Execution:

Followup prompt:
“As I move the mouse over a grid square it should highlight the square in red”

Execution:

This is usable right now. While it can’t — yet — write larger programs, it’s absolutely a quick way to write parts that can be taken and expanded on, and it’s a super easy way to test concepts.

Exciting times.

More ChatGPT posts by me on Twitter and Mastodon

Edit 03.03.2023: Pimoroni has updated the firmware to include numpy. There’s still useful info below on custom firmware :)

I bought an excellent Galactic Unicorn from Pimoroni, with the aim of connecting a Time-of-Flight sensor (VL53L5CX) to use as input for the board, connecting it over the QW/ST using I2C.

Sounds simple right? No.

There are some examples for the VL53L5CX working with the Raspberry Pi Pico, so I assumed this would work OK with the Galactic Unicorn, which uses a Pico W (A Pico with WiFi, great for IoT!).

Was I right? Also no.

The example I needed for the VL53L5CX used ulab numpy, a module created to speed up mathematical operations on arrays, and it turns out this isn’t included in the Galactic Unicorn firmware. It’s quite a large module, so has been removed to save space in a number of the Pimoroni uf2’s.

With some advice from Jimmo on the Micropython Discord, and some pointers from Gadgetoid on Discord and Mastodon, I was able to remove a bunch of modules from the Galactic Unicorn firmware, and add in ulab numpy.

How to make custom Pimoroni firmware with ulab numpy

If you want to make your own custom firmware, turns out it’s pretty easy. It requires a bit of github knowledge. I’ll outline the steps:

1. Go to https://github.com/pimoroni/pimoroni-pico and fork it.
2. In your new repo, create a branch
3. Go to micropython/modules/micropython-picow_galactic_unicorn.cmake
4. Under #Sensors & Breakouts, comment out everything you don’t need. This saves some space, it might not be necessary, but as I didn’t need support for anything except the VL53L5CX, I removed them all
5. After #include(micropython-common) add:

function(enable_ulab)
    include(ulab/code/micropython)

target_compile_definitions(usermod_ulab INTERFACE
        # Support for complex ndarrays
        ULAB_SUPPORTS_COMPLEX=0

# Determines, whether scipy is defined in ulab. The sub-modules and functions
        # of scipy have to be defined separately
        ULAB_HAS_SCIPY=0

# The maximum number of dimensions the firmware should be able to support
        # Possible values lie between 1, and 4, inclusive
        ULAB_MAX_DIMS=2

# By setting this constant to 1, iteration over array dimensions will be implemented
        # as a function (ndarray_rewind_array), instead of writing out the loops in macros
        # This reduces firmware size at the expense of speed
        ULAB_HAS_FUNCTION_ITERATOR=1

# If NDARRAY_IS_ITERABLE is 1, the ndarray object defines its own iterator function
        # This option saves approx. 250 bytes of flash space
        NDARRAY_IS_ITERABLE=1

# Slicing can be switched off by setting this variable to 0
        NDARRAY_IS_SLICEABLE=1

# The default threshold for pretty printing. These variables can be overwritten
        # at run-time via the set_printoptions() function
        ULAB_HAS_PRINTOPTIONS=1
        NDARRAY_PRINT_THRESHOLD=10
        NDARRAY_PRINT_EDGEITEMS=3

# determines, whether the dtype is an object, or simply a character
        # the object implementation is numpythonic, but requires more space
        ULAB_HAS_DTYPE_OBJECT=0

# the ndarray binary operators
        NDARRAY_HAS_BINARY_OPS=0

# Firmware size can be reduced at the expense of speed by using function
        # pointers in iterations. For each operator, the function pointer saves around
        # 2 kB in the two-dimensional case, and around 4 kB in the four-dimensional case.

NDARRAY_BINARY_USES_FUN_POINTER=1

NDARRAY_HAS_BINARY_OP_ADD=1
        NDARRAY_HAS_BINARY_OP_EQUAL=1
        NDARRAY_HAS_BINARY_OP_LESS=1
        NDARRAY_HAS_BINARY_OP_LESS_EQUAL=1
        NDARRAY_HAS_BINARY_OP_MORE=1
        NDARRAY_HAS_BINARY_OP_MORE_EQUAL=1
        NDARRAY_HAS_BINARY_OP_MULTIPLY=1
        NDARRAY_HAS_BINARY_OP_NOT_EQUAL=1
        NDARRAY_HAS_BINARY_OP_POWER=1
        NDARRAY_HAS_BINARY_OP_SUBTRACT=1
        NDARRAY_HAS_BINARY_OP_TRUE_DIVIDE=1
    )
endfunction()
enable_ulab()

• Now create a release, and commit & push your changes.
• Github Actions will now compile everything. You need to wait a short while, and you’ll find your UF2 in Actions > Micropython (second one down) > select your latest Workflow Run at the top, then scroll down, and you’ll see your firmware with ulab included! Eg: pimoroni-picow_galactic_unicorn-[blah]-micropython.uf2

You can in theory do the same for the other boards, take a look in /micropython/modules, and view all the various cmake files.

Hope that quick overview is helpful to someone!

Upgrading from the XS Max, I didn’t think the 13 Pro Max would be much bigger – the specs show the dimensions are almost the same – but the flat sides make a big difference. It feels noticeably chunkier, and is definitely not as comfortable to hold.

People seem to be gushing over the 120hz screen, but I honestly don’t notice much difference. Perhaps as iOS is so smooth and responsive anyway, I don’t really see the difference? Or maybe my eyes are broken.

The LiDAR scanner is incredible. Using Polycam to build real-time meshes is quite a sight to behold. I recognise this is pretty niche, but it’s my niche. If it’s of interest to you, you’ll probably love it.

One of the main reasons I wanted the upgrade is for a better camera, and I’m very happy with the quality of the images, but the camera App is becoming a bit of a disorganised mess. Apple can do much better here. I had to google how to use various functions, and that’s a bit disappointing.

Apple’s silicon cover is a bit slippery on such an unwieldy phone. I’ll keep it, but will probably get something with a bit more grip in a few months.

The increase in weight – almost quarter of a kilo now – does make me worry that if I’m in bed using my phone and accidentally drop it on my face, it’ll break my nose.

Overall, it’s an impressive bit of kit. I do like it, but the UX of the camera App and the sheer size is a bit annoying.

See my Instagram for some sample macros.

Update

With a magsafe popgrip the Max becomes a lot more usable. Bit steep at £30.

Aka Facebook privacy nightmare shades.

Most of my day, I have an Oculus Quest 2 on my head, so obviously I have surrendered my soul to Facebook already. Most reviews seem to cover privacy concerns, so I’m going to ignore that. Go read those if you’re interested in that — and you should be.

I have several pairs of Ray-Bans, but no Wayfarers, so went for those, in blue, which make the cameras slightly more noticeable. This wasn’t intentional, I thought I bought black ones. Oh well. They fit well, slightly heavier than normal Ray-bans but I didn’t notice it after a few minutes.

Speakers are better than I thought they’d be, great for phone calls, videos, and podcasts, but not really music. Also anyone around you can hear it pretty clearly.

The mics work really well for phone calls and speech-to-text, as well as audio for video. Vaguely similar to AirPods Pro with Transparency on, except not as good but more convenient.

Touchpad controls for volume on the right arm seem a bit hit and miss.

The shutter button is in exactly the place I hold shades while putting them on and taking them off so I accidentally hit record every time.

Facebook assistant works well — I was going to disable it, but it’s so convenient to use. Dammit.

When you want to get photos and videos off the device, the app needs to connect to your shades via a Wi-fi network. Doesn’t automatically transfer media to your phone. It is pretty seamless as-is though. Once the media is transferred, it stays in the View app where you can edit it and share to other apps as required. It doesn’t automatically upload to Facebook.

The shades are a super easy way to capture media (and consume audio) without context switching to another device. I really really like it. The sacrifice in quality is worth it for the convenience.

In time, over future iterations, smart glasses like these will get packed with more and more tech, and as I didn’t get Google Glass, Snap Spectacles, or Bose Frames, I thought I’d jump on board here to see what these devices are like, and I’m pretty impressed so far.

Improvements

The single LED for recording isn’t enough IMO. A larger area is needed to make it more obvious. A light bar along the top of the shades maybe? Holding up a slab of glass and metal to take photos is quite an obvious postural thing to do, and noticeable by people around you. There needs to be some kind of equivalent here.

The speakers could be bone conductive instead, so not everyone can hear what you’re listening to.

Reposition the shutter button so it’s not exactly where you hold shades when putting them on.

Touchpad controls need to work reliably.

Sample montage from the View App

https://www.instagram.com/p/CTxZ_5QqgXn/

Update — a few days later

When I stop wearing them, I find myself missing them. Mainly for the convenience of the mic and speakers when using my phone.

Update 2

I was recorded by my own Ray-Ban Stories, and this was my reaction

https://twitter.com/yezzer/status/1438917690115117059

Iestyn is CTO at Future Visual, an award-winning VR & AR studio based in Brighton, UK that works with enterprise clients across the globe.

Future Visual develops ground-breaking immersive experiences that enable users to access situations and scenarios that are either physically impossible or prohibitively expensive to recreate in the real world.

Ray-Ban Stories — notes was originally published in Virtual Library on Medium, where people are continuing the conversation by highlighting and responding to this story.

[Moving from my other account]

For anyone who’s wondering how to save the meshes generated by 6d.ai, it’s pretty straightforward. I’ve seen a few people asking and no definitive answer, so thought I’d write it up in case anyone searching for this found it useful.

The steps are basically:

1. Combine the chunks into one mesh.
   I used Unity’s built-in Combine Meshes. You can see a sample of the code here, which I adapted.
2. Save the mesh locally.
   I used obj-unity3d to save the mesh as an obj to Application.persistentDataPath.
3. Access the mesh.
   Currently 6d.ai is iOS only, so plug your iOS device in, open XCode, and do the following:
   Window>Devices and Simulators>Select your App> Press the cog>Download container > Save > Navigate to the container >Right click >Show package contents>App Data>Documents. Your file will be there!
   You could also upload the local OBJ to a cloud storage solution for easier access. I did this using Firebase, but that’s a post for another time.

A hacky code sample that might help you out:

//Get a reference to the SDMesh
 SDMesh sdMesh = FindObjectOfType<SDMesh>();
 
 //combine the meshes
 MeshFilter[] meshFilters = sdMesh.GetComponentsInChildren<MeshFilter>();
 CombineInstance[] combine = new CombineInstance[meshFilters.Length];
 int i = 0;
 while (i < meshFilters.Length)
 {
 Debug.Log("combining.."+meshFilters[i].gameObject.transform.name);
 combine[i].mesh = meshFilters[i].sharedMesh;
 combine[i].transform = meshFilters[i].transform.localToWorldMatrix;
 i++;
 }
 Mesh combinedMesh = new Mesh();
 combinedMesh.CombineMeshes(combine);
 
 //save the mesh
 OBJData data = combinedMesh.EncodeOBJ();
 string filepath = Application.persistentDataPath + "/Obj_" + UnityEngine.Random.Range(0, 10000) + ".obj";
 var lStream = new FileStream(filepath, FileMode.Create);
 OBJLoader.ExportOBJ(data, lStream);
 lStream.Close();

Originally published at https://medium.com on November 24, 2018.