fix sys_write logger output

[defunctio]

• write/writev/read bugfixes
• add typing annotations
• resolves Odd logger output for linux logger #1020
• resolves Incorrect argument width being used with some syscalls #940

This change is 

[offlinemark]

what's the reason for latin-1?

[defunctio]

latin-1 will decode 0x00-0xff

[offlinemark]

regular line.decode() seems to work fine for me. tbh i don't know a whole lot about encodings, is there any reason we shouldn't just use utf-8?

[defunctio]

utf-8 it would raise with UnicodeDecodeError on any byte > 0x7f and in this case what is being written is just binary data. would be nice to replace this at a later date with something similar of xxd output. but that might be too verbose

[offlinemark]

>>> b'sdkjfhk\x20\x8f'.decode()
Traceback (most recent call last):
  File "<input>", line 1, in <module>
    b'sdkjfhk\x20\x8f'.decode()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8f in position 8: invalid start byte
>>> b'sdkjfhk\x20\x8f'.decode('latin-1')
'sdkjfhk \x8f'
>>>

[offlinemark]

data can be a list of bytes, but it can also contain Expression objects (ArraySelect) instances, which i believe is why the str() call was there. so we i think we still need to do something like str(x).encode()

one way to easily set this up:

import manticore
m = manticore.Manticore('/bin/ls')
s = m.initial_state
cpu = s.cpu
arr = s.new_symbolic_buffer(4)
cpu.push_bytes(arr)
cpu.read_bytes(cpu.RSP, 4)
b = cpu.read_bytes(cpu.RSP, 4)

[defunctio]

won't data = self._transform_write_data(data) line 1292 directly above only return concrete data?

Also, seems there is a bug in SLinux._transform_write_data where concrete_data should be bytes()

[offlinemark]

won't data = self._transform_write_data(data) line 1292 directly above only return concrete data

ah you're totally right, sorry about that.

Also, seems there is a bug in SLinux._transform_write_data where concrete_data should be bytes()

perhaps it should be bytes, i think these functions were all generally written to take/return lists. for example when write_fd.write(data) happens, File.write is written to loop over the argument, rather than just passing it into file.write().

[offlinemark]

since data is guaranteed to be concrete, instead of b''.join([x for x in data]) i think we can do b''.join(data)

[defunctio]

data should already guaranteed to be bytes we can just use data.split(b'\n')
This did lead me to noticing there was an issue with SLinux.sys_writev also though.

I should probably review / test all the read/write syscalls to see if anything else shows up.

[offlinemark]

data should already guaranteed to be bytes we can just use data.split(b'\n')

i think data is a list of bytes, not bytes

I should probably review / test all the read/write syscalls to see if anything else shows up

this would be amazing 🙏

[defunctio]

i think data is a list of bytes, not bytes

Ah... local changes I already made I've switched this to using bytes. Any objection to that? There are very few references that use it internally and this just seems to make more sense.

[defunctio]

updated and tested read/readv/write/writev, everything appears to be working as expected

[offlinemark]

Reviewed 2 of 3 files at r2.
Reviewable status: 2 of 3 files reviewed, 4 unresolved discussions (waiting on @Mossberg and @defunctio)

---

manticore/platforms/linux.py, line 12 at r2 (raw file):

import binascii

# Remove in favor of binary.py

this comment should stay close to the elffile imports which it is meant for

---

manticore/platforms/linux.py, line 1306 at r2 (raw file):

            data: MixedSymbolicBuffer = cpu.read_bytes(buf, count)
            data: bytes = self._transform_write_data(data)
            write_fd.write(data)

File.write iterates over the argument calling file_object.write(c) for each. Since data is a bytes now, calling file_object.write() will fail i think since it's not meant to take an int

---

manticore/platforms/linux.py, line 1313 at r2 (raw file):

                             buf,
                             count,
                             line.decode('latin-1'))

could we actually add a little comment here on the latin line explaining it? i think it is a little non-obvious, and warrants a small comment

[defunctio]

Reviewable status: 2 of 3 files reviewed, 4 unresolved discussions (waiting on @Mossberg and @defunctio)

---

manticore/platforms/linux.py, line 12 at r2 (raw file):

Previously, mossberg (Mark Mossberg) wrote…

this comment should stay close to the elffile imports which it is meant for

Done.

---

manticore/platforms/linux.py, line 1306 at r2 (raw file):

Previously, mossberg (Mark Mossberg) wrote…

File.write iterates over the argument calling file_object.write(c) for each. Since data is a bytes now, calling file_object.write() will fail i think since it's not meant to take an int

ack, you are correct indeed. i was using stdin/stdout in my tests which is handled bySocket

---

manticore/platforms/linux.py, line 1313 at r2 (raw file):

Previously, mossberg (Mark Mossberg) wrote…

could we actually add a little comment here on the latin line explaining it? i think it is a little non-obvious, and warrants a small comment

Done.

[offlinemark]

i wonder, for lines like data: bytes = bytes() can mypy and similar infer the type of data? do they?

also have you tried running mypy/etc on this with the annotations?

lgtm

Reviewed 1 of 1 files at r3.
Reviewable status: complete! all files reviewed, all discussions resolved