captainsafia

[captainsafia]

Advent of Code Day 3

Another day, another Advent of Code challenge! As per usual, you can read the problem statement for today's challenge here.

I'm solving each of these problems using a different programming language. I've already used Python and JavaScript and have wondered what language I can use next. It's a little bit later in the day as I'm working on this and I've been working all day and don't have the mental capacity to try a new language. So for this challenge, I'm breaking the rule and working with a programming language I've already used in these challenges: Python.

So, in this challenge, we'd like to determine the area of overlapping assignments given a list of positions and dimensions. The first thing I wanted to do here was to write a function that would determine all the child coordinates within the range of a rectangle.

def increments(claim): return [ (x, y) for x in range(claim["x"], claim["x"] + claim["width"]) for y in range(claim["y"], claim["y"] + claim["height"]) ]

The next thing I needed to do was parse the input file. Although I usually stay far away from it, I decided to use a regular expression to parse the inputs. I copy-pasted some of the data into Regexr and played around with the values until I eventually got a regular expression to parse things out. Finally, I set up an iterator to run through each inch increment within a rectangle and increase the value of the overlapped area by 1. Running through the areas inch by inch allows us to to only count overlaps that involve multiple rectangles only once.

For the second part of the puzzle, I needed to determine the ID of the claim that did not share an area with any other claims. I did this by searching through each inch of each claim for the areas that were not shared. Here's what the end result looked like.

import re def increments(claim): return [ (x, y) for x in range(claim["x"], claim["x"] + claim["width"]) for y in range(claim["y"], claim["y"] + claim["height"]) ] with open("areas.txt") as claims: regex = "^#(\d+)\s@\s(\d+),(\d+):\s(\d+)x(\d+)" parsed_claims = [] overlap = 0 for claim in claims.readlines(): m = re.search(regex, claim) parsed_claims.append( { "id": m.group(1), "x": int(m.group(2)), "y": int(m.group(3)), "width": int(m.group(4)), "height": int(m.group(5)), } ) overlaps = {} for claim in parsed_claims: for increment in increments(claim): if increment in overlaps: overlaps[increment] += 1 else: overlaps[increment] = 1 overlapped_area = 0 for overlap in overlaps.values(): if overlap > 1: overlapped_area += 1 print(overlapped_area) for claim in parsed_claims: if all(overlaps[increment] == 1 for increment in increments(claim)): print(claim)

[captainsafia]

Eh, ship it!

It's Day 2 of the Advent of Code 2018 Challenge. Once again, I'm blogging through the solution here. Before getting started, you should read the day's challenge.

I started off by downloading the list of box IDs into a box-ids.txt file. The general program will require that we run through the IDs, keeping a count of the times we find two and three of the same letter in the ID, and then multiplying these counts.

Since I've decided to do the challenge in a different language each day. Today, I'll be writing out the solution in Node. The Node standard library isn't as rich as Python's standard library. There are some standard libraries in Python that would be extremely applicable to this problem, but oh well. I dumped out the initial iteration of the algorithm above here.

const fs = require("fs"); function getOccurenceCount(string) { let result = {}; for (var index in string) { const char = string.charAt(index); result[char] = (result[char] || 0) + 1; } return result; } fs.readFile("box-ids.txt", (error, boxIdsBuffer) => { if (error) console.error(error); let countOfTwos = 0; let countOfThrees = 0; const boxIds = boxIdsBuffer.toString().split("\n"); for (let index in boxIds) { const boxId = boxIds[index]; occurenceCount = getOccurenceCount(boxId); occurenceCountList = Object.values(occurenceCount); countOfTwos += occurenceCountList.includes(2); countOfThrees += occurenceCountList.includes(3); } console.log(countOfThrees * countOfTwos); });

Fun fact, it actually took me about 5 minutes extra to get this initial iteration because I didn't fully read the instructions (shame on me!). I assumed you had to add the number of times a character occurred twice, not just if a character occurred twice. That sentence was phrased awkwardly, but the copy and paste from the problem statement should hopefully clarify what I mean.

abcdef contains no letters that appear exactly two or three times. bababc contains two a and three b, so it counts for both. abbcde contains two b, but no letter appears exactly three times. abcccd contains three c, but no letter appears exactly two times. aabcdd contains two a and two d, but it only counts once. abcdee contains two e. ababab contains three a and three b, but it only counts once.

As it turns out, there is a second part to this particular puzzle. I'll post the link for it here. I'm not sure if you are going to be able to see it if you haven't completed the first part. I'll work on the second part now and then optimize both solutions later, because trust me, the above could definitely use a little love in some places.

For this second part, we will need to find the IDs that differ by exactly a single letter. I'm going to bet that I'll need to compute the Hamming distance between the password strings at some point.

Intermission: It's been about 8 hours since I wrote the last sentence. I had to go and be a human being so now I'm back rushing to finish this. Here's the naive solution I cooked up for calculating the two passwords that have only a single string difference.

const fs = require("fs"); function getOccurenceCount(string) { let result = {}; for (var index in string) { const char = string.charAt(index); result[char] = (result[char] || 0) + 1; } return result; } function hammingDistance(s, t) { let distance = 0; for (let index in s) { if (s[index] !== t[index]) { distance += 1; } } return distance; } fs.readFile("box-ids.txt", (error, boxIdsBuffer) => { if (error) console.error(error); let countOfTwos = 0; let countOfThrees = 0; const boxIds = boxIdsBuffer.toString().split("\n"); for (let index in boxIds) { const boxId = boxIds[index]; occurenceCount = getOccurenceCount(boxId); occurenceCountList = Object.values(occurenceCount); countOfTwos += occurenceCountList.includes(2); countOfThrees += occurenceCountList.includes(3); } console.log(countOfThrees * countOfTwos); for (let index in boxIds) { const boxId = boxIds[index]; boxIds.map(otherBoxId => { if (hammingDistance(boxId, otherBoxId) === 1) { for (let index in boxId) { if (boxId[index] === otherBoxId[index]) { process.stdout.write(boxId[index]); } } return; } }); } });

Alright! That gets the job done. There's a couple of ways to make this better. Using streams instance of converting the buffer to a string and iterating over it, reducing the number of repetitive for-loops, cleaning up the calculation for the number of occurrences.

Here's the final clean-up I did. Eh, it does the trick. Ship it!

const fs = require("fs"); function getOccurenceCount(string) { let result = {}; for (var index in string) { const char = string.charAt(index); result[char] = (result[char] || 0) + 1; } return result; } function hammingDistance(s, t) { let distance = 0; for (let index in s) { if (s[index] !== t[index]) { distance += 1; } } return distance; } fs.readFile("box-ids.txt", (error, boxIdsBuffer) => { if (error) console.error(error); let countOfTwos = 0; let countOfThrees = 0; const boxIds = boxIdsBuffer.toString().split("\n"); for (let index in boxIds) { const boxId = boxIds[index]; occurenceCount = getOccurenceCount(boxId); occurenceCountList = Object.values(occurenceCount); countOfTwos += occurenceCountList.includes(2); countOfThrees += occurenceCountList.includes(3); boxIds.map(otherBoxId => { if (hammingDistance(boxId, otherBoxId) === 1) { console.log( Array.from(boxId) .filter((character, index) => { return character === otherBoxId[index]; }) .join("") ); } }); } console.log(countOfThrees * countOfTwos); });

[captainsafia]

Oh gosh, I’m blogging again

dusts off shoulders

So it's been a while since I've done this.

That's right, I'm blogging again. Prepare your food rations and gallons of water because a disaster is coming through!

In all seriousness, I won't joke too much about me getting back on the old blogging steed.

Since it's officially December 1st, which constantly scares me because I was not expecting it to come this fast, the Advent of Code has started.

I figured that I would play along this year. I haven't done it before, but I'm no stranger to solving the odd programming puzzle or two. I figured I'd start blogging along the experience of me solving these problems, in real-time. I've also been looking for a way to pick up some new programming languages. So, I decided that I'll be doing each day of the challenge using a different programming language.

It's the first day so the text for the challenge has been released. You can read the day 1 challenge here. Be sure to read it before continuing with the rest of the blog post.

Alright. Ready? Let's go.

The fundamental problem here is that given a newline-delimited file with a list of numbers with "+" or "-" we want to figure out what the resulting number will be after all the numbers have been summed or subtracted, starting from 0.

I saved the data into a file called frequencies.txt and I'll be using it throughout the code. For this first challenge, I'll start with a language that is familiar and well-worn to me: Python. It was the first programming language (other than HTML and CSS) that I had learned as a pre-teen. It feels fitting to use it here.

I started by implementing a pretty naive solution. I read the file into a list data structure, looped through the strings, checked to see if the string started with "-" or "+" add executed the following computation. Here's what the code for this looks like.

with open("frequencies.txt") as frequencies: result = 0 numbers = frequencies.read().splitlines() for number in numbers: if number.startswith("-"): number_as_int = int(number[1:]) result -= number_as_int elif number.startswith("+"): number_as_int = int(number[1:]) result += number_as_int else: raise Exception("{} is not in the valid format.".format(number)) print(result)

This code is correct, I validated the result I got against the Advent of Code page to confirm this. Is there a way we can make it better?

Now, this is usually the part in these blog posts where someone figures out some clever optimization for the code or uses some special language features to reduce the lines of code. As an open source developer and avid code reader, I always prefer to keep code readable. Any change I make to the code would have to maintain the readability of the code.

As it turns out, I can get away with removing the .startswith logic in the codebase. Python's int function can handle converting numeric strings with any sign. So it'll appropriately handle converting "+2" and "-3". Knowing this, the code can be trimmed down to the code below.

with open("frequencies.txt") as frequencies: result = 0 numbers = frequencies.read().splitlines() for number in numbers: result += int(number) print(result)

Well, isn't this neat! It's still fairly readable. You don't miss the point of the code by skimming through it really quickly. I like that.

I could leverage another Python feature here, the list comprehension, to make the code even more succinct. Here's how it would look.

with open("frequencies.txt") as frequencies: result = sum([int(number) for number in frequencies.read().splitlines()]) print(result)

Now, this is where I personally draw a line. Does this code get the job done? Yes. Is it nice and short? Yes. Is it easy to get a sense of what is going on here from a quick skim? Ehhhhhhhhhhh.

If I were to chose the iteration I would choose to implement, I'd probably choose the second. It balances white space and variable naming easily and doesn't overdo it with using language-specific features.

Someone who has never read Python before but has experience programming can under the second iteration quite easily, not so much the second. I think this is one of the most important things about good code: it should be easy to read and understand as long as someone understands the basic concepts of programming.

OK. That's enough preaching for one day. Hopefully, you enjoyed reading this. If you'd like to see the solutions and try them out yourself, you can check out this GitHub repo.

See you tomorrow!

[captainsafia]

Writing to win friends and influence people

Communication is important. This is a point so obvious and indisputable that I won't dive into here. Being able to communicate effectively with romantic partners, co-workers, family members, contractors, doctors, and even the people you're not in the mood to talk to is key to existing as a person on the planet.

I don't think anyone has mastered the art of communicating all the time effectively. Mostly because the measure of effectiveness is subjective, but also because it's hard.

At a fundamental level, communication is all about building a connection with the people you are speaking with. Easier said than done. I spent some time thinking about the writing techniques that I use in business and personal writing to connect more deeply with the people reading my writing.

1. Don't start sentences with a verb, start with a pronoun.

Pronouns are cheap. They only cost you a couple of extra letters, but they go a long way into making your terse messages seem personable. When you start a sentence with a pronoun ("Delete this line of code." vs. "You can delete this line of code."), you make the statement seem less like a condescending command and also empower and center the reader of the statement. Empowered readers are happy readers!

2. Use a single sentence to describe the context, use links and references for anything extra.

One of the hardest parts of conversing in a work setting is making sure that the person you are talking to understands the context behind what you are trying to communicate. Context is things like the history of a long-standing bug in the product, information about a client's needs, and so on. I tend to be the kind of person who spends a lot of time re-explaining the context. I'm trying to shift to a more effective strategy where I provide the context in a single sentence and links elsewhere. For example,

"This pesky bug is a result of the plotting library we use (link to details), but we've discussed several possible solutions (link to details)."

A neat side-effect of this is that it forces you to actively document details so that you can easily reference them later.

3. Formulate a request as a question.

This is a pretty popular technique that you might be familiar with. Instead of saying "Call me at 2 pm." Ask "Can you call me today at 2 pm?" Why does it sound so much nicer? For the same reason #1 does. It centers the reader and gives them power and agency. And again, empowered readers are happy readers!

4. Share your state of mind with the reader.

Life isn't always perfect. Maybe you got stuck in horrible traffic this morning. Maybe your coffee machine broke (I'm so sorry). Maybe you're having trouble with family. Your state of mind affects the way you write. If you trust the person reading your communication, let them know what's going on. Preface your email or comment with a "Writing this in a crowded train" or "I haven't had my coffee yet but all this is with good intent." In addition to clarifying the emotional intent, messages like these also build a sense of trust and intimacy between you and the reader.

5. Don't write in haste.

This tip seems rather obvious. Although your writing may be brief, the time you take to write it should not be. Once you hit the Enter key, you can't take those words back so don't write something you'll regret.

And that's all I've got! Hopefully, you picked up on the general theme of these tips: be intentional about the words you use and how people might interpret them. It'll leave an impact way beyond what you wrote.

Do you have any tips for communicating politely but tersely in your work life? Let me know on Twitter!

[captainsafia]

A purposeful hiatus

It's the end of the week.

To be honest, I'm getting a little tired of reading through straces. And also reading through code bases. Maybe it's just the weariness of working full-time and hustling on Zarf. Who knew that the most difficult thing about finally regularly blogging was running out of ideas?

So, I figure, I'll just free-form this and write whatever is on my mind.

Right now, it's ideas. I find the cultural fascination with ideas, well, fascinating. It's closely tied to the fascination with creativity, but not just any creativity, productive creativity. The kind that you would be OK within a capitalist society. There's very little room for pointless creativity in today's society.

If you're going to think of something, it has to make you money, or "advance your career," or save the world in some way. I'm not saying that those things aren't valid pursuits, they just make me feel a little claustrophobic sometimes.

I think this might relate to the "tired" feeling that I mentioned earlier. I've had a few ideas over the past couple of weeks but none of them that feel...productive. But what is productivity, after all, if not an arbitrary and immeasurable construct? Heck, at what point did I lose my spark and decide to pathologically smother ideas that didn't mean some ill-defined measure of productivity?

Since publishing Exits, I've worked on a few fiction pieces but nothing that has yet to materialize. I've been publishing blog posts on here, but I'll admit that they are not as good as I would like them to be. What have I been writing a lot of lately? Emails and code and more emails.

When I first started this journey of consistently blogging, I was looking to get better at writing. Now that I've gotten a grasp of that, I find that the difficult thing to do is figure out what to write. I've had a fair bit of luck solving this problem for my bi-weekly newsletter by implementing reader polling to determine what topics I should write about.

So I guess that this is what this blog post is about, me running out of ideas. At this point, I'm willing to take the mature route and say that I'll write only when I have something to say, not based on some arbitrary schedule.

So I guess see you whenever...

[captainsafia]

Looking into curl: part 2

Is it Wednesday already? I guess it is. To be honest, I pretty much write these blog posts on auto-pilot, so the time passes by really quickly.

I'm continuing the series that I started on Monday where I analyze the strace for curl. In the last blog post, I looked into how the curl process managed domain name resolution. I'll continue from there in this blog post.

The next chunk of the trace looks like the following.

Sidebar: I dunno why I say "looks like," this is exactly what it is. Alas, inner monologues lack precision.

socket(AF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) = 3 [pid 8819] connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("169.254.169.254")}, 16) = 0 [pid 8819] poll([{fd=3, events=POLLOUT}], 1, 0) = 1 ([{fd=3, revents=POLLOUT}]) [pid 8819] sendmmsg(3, [{msg_hdr={msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\252_\1\0\0\1\0\0\0\0\0\0\5safia\5rocks\0\0\1\0\1", iov_len=29}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, msg_len=29}, {msg_hdr={msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\375\266\1\0\0\1\0\0\0\0\0\0\5safia\5rocks\0\0\34\0\1", iov_len=29}], msg_iovlen=1, msg_controllen=0, msg_flags=MSG_EOR|MSG_CONFIRM|MSG_RST|MSG_MORE|MSG_WAITFORONE|MSG_CMSG_CLOEXEC|0x80200000}, msg_len=29}], 2, MSG_NOSIGNAL) = 2 [pid 8819] poll([{fd=3, events=POLLIN}], 1, 5000 <unfinished ...> [pid 8818] <... poll resumed> ) = 0 (Timeout)

The first system call initiates a connection to an IP address. The IP address wasn't giving me any clues, but I thought the port number it was connecting to was interesting: port 53. I looked this up, and as it turns out, port 53 is the default port number used by DNS servers. It appears that the connection it is making her is to a DNS server in an attempt to resolve the domain name. This suspicion was confirmed by the contents of the payload that is sent via the sendmmsg command later down the line.

[pid 8819] ioctl(3, FIONREAD, [85]) = 0 [pid 8819] recvfrom(3, "\375\266\201\200\0\1\0\2\0\0\0\0\5safia\5rocks\0\0\34\0\1\300\f\0"..., 2048, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("169.254.169.254")}, [28->16]) = 85 [pid 8819] poll([{fd=3, events=POLLIN}], 1, 4940) = 1 ([{fd=3, revents=POLLIN}]) [pid 8819] ioctl(3, FIONREAD, [61]) = 0 [pid 8819] recvfrom(3, "\252_\201\200\0\1\0\2\0\0\0\0\5safia\5rocks\0\0\1\0\1\300\f\0"..., 65536, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("169.254.169.254")}, [28->16]) = 61 [pid 8819] close(3)

The next chunk of the stack trace reads data from some gai.conf file. I decided to look up what it might be responsible for in the docs. gai.conf is a file that stores the configuration for the getaddrinfo function. The getaddrinfo function is responsible for a lot of the heavy lifting when it comes to mapping a hostname to an IP address that we can connect to. The configurations stored in gai.conf deal with what should be done when a call to getaddrinfo returns multiple responses. I assume this means that when a hostname resolves to multiple addresses. In this case, the gai.conf file allows us to persist information about which address we actually want to connect to.

pid 8819] open("/etc/gai.conf", O_RDONLY|O_CLOEXEC) = 3 [pid 8819] fstat(3, {st_mode=S_IFREG|0644, st_size=2584, ...}) = 0 [pid 8819] fstat(3, {st_mode=S_IFREG|0644, st_size=2584, ...}) = 0 [pid 8819] read(3, "# Configuration for getaddrinfo("..., 4096) = 2584 [pid 8819] read(3, "", 4096) = 0 [pid 8819] close(3)

The next chunk of the stack trace was a little it more esoteric and difficult for me to parse. I noticed that the program was opening a new socket with an AF_NETLINK type. After researching this, I discovered that this particular connection type allows the kernel to communicate with the userspace. In this case, the socket isn't facilitating a connection with the outside world. Unfortunately, knowing this doesn't give me much context into what is actually being communicated over this socket. I'm making a mental to-do to go one layer up and see what looking through the curl code base again might tell me.

[pid 8819] futex(0x7f21c8972ee4, FUTEX_WAKE_PRIVATE, 2147483647) = 0 [pid 8819] socket(AF_NETLINK, SOCK_RAW, NETLINK_ROUTE) = 3 [pid 8819] bind(3, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, 12) = 0 [pid 8819] getsockname(3, {sa_family=AF_NETLINK, nl_pid=8818, nl_groups=00000000}, [12]) = 0 [pid 8819] sendto(3, {{len=20, type=0x16 /* NLMSG_??? */, flags=NLM_F_REQUEST|0x300, seq=1526219697, pid=0}, "\0\0\0\0"}, 20, 0, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, 12) = 20 [pid 8819] recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=[{{len=76, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1526219697, pid=8818}, "\2\10\200\376\1\0\0\0\10\0\1\0\177\0\0\1\10\0\2\0\177\0\0\1\7\0\3\0lo\0\0"...}, {{len=88, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1526219697, pid=8818}, "\2 \200\0\2\0\0\0\10\0\1\0\n\200\0\2\10\0\2\0\n\200\0\2\10\0\4\0\n\200\0\2"...}, {{len=0, type=0 /* NLMSG_??? */, flags=0, seq=0, pid=0}}], iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 164 [pid 8819] recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=[{{len=72, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1526219697, pid=8818}, "\n\200\200\376\1\0\0\0\24\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\24\0\6\0"...}, {{len=72, type=0x14 /* NLMSG_??? */, flags=NLM_F_MULTI, seq=1526219697, pid=8818}, "\n@\200\375\2\0\0\0\24\0\1\0\376\200\0\0\0\0\0\0@\1\n\377\376\200\0\2\24\0\6\0"...}, {{len=393236, type=0xffff /* NLMSG_??? */, flags=NLM_F_REQUEST|NLM_F_MULTI|NLM_F_ACK|NLM_F_ECHO|NLM_F_DUMP_INTR|NLM_F_DUMP_FILTERED|0xffc0, seq=4294967295, pid=282}, "\32\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"...}], iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 144 [pid 8819] recvmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=[{{len=20, type=NLMSG_DONE, flags=NLM_F_MULTI, seq=1526219697, pid=8818}, "\0\0\0\0"}, {{len=1, type=0x14 /* NLMSG_??? */, flags=NLM_F_REQUEST, seq=0, pid=0}}], iov_len=4096}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 20 [pid 8819] close(3)

The next chunk of the trace is equally as esoteric for me. In this case, the socket connection made communicates with an IPv6 address. Again, completely clueless as to what is going on here. In addition to reading the code for curl, I think running Wireshark and looking at what the network dump tells me might illuminate more here. In any case, I'm not all that interested in getting into the nitty-gritty of what is going on here. It seems pretty clear that curl goes through quite a few steps to resolve a domain name from both local and remote DNS services.

[pid 8819] socket(AF_INET6, SOCK_DGRAM, IPPROTO_IP) = 3 [pid 8819] connect(3, {sa_family=AF_INET6, sin6_port=htons(443), inet_pton(AF_INET6, "2400:cb00:2048:1::681b:a05e", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, 28) = -1 ENETUNREACH (Network is unreachable) [pid 8819] connect(3, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0 [pid 8819] connect(3, {sa_family=AF_INET6, sin6_port=htons(443), inet_pton(AF_INET6, "2400:cb00:2048:1::681b:a15e", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, 28) = -1 ENETUNREACH (Network is unreachable) [pid 8819] connect(3, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0 [pid 8819] connect(3, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("104.27.160.94")}, 16) = 0 [pid 8819] getsockname(3, {sa_family=AF_INET6, sin6_port=htons(56904), inet_pton(AF_INET6, "::ffff:10.128.0.2", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, [28]) = 0 [pid 8819] connect(3, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0 [pid 8819] connect(3, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("104.27.161.94")}, 16) = 0 [pid 8819] getsockname(3, {sa_family=AF_INET6, sin6_port=htons(43230), inet_pton(AF_INET6, "::ffff:10.128.0.2", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, [28]) = 0 [pid 8819] close(3) = 0

At this point, the process (thread) with PID 8819 exits and we return back to the main thread. I'll start looking at that portion of the stack trace in the next blog post.

[captainsafia]

Looking at the curl stack trace: part 1

As mentioned in my last blog post, I'm hoping to do a thorough analysis of the strace for curl in the next couple of blog posts. I started looking through the stack trace in the last blog post, but I quickly realized that it would take more than one blog post to look through everything so here I am.

Alright, here's the first chunk of the stack trace that I'll be looking through.

[pid 8819] set_robust_list(0x7f21c40f09e0, 24) = 0 [pid 8819] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3 [pid 8819] connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory) [pid 8819] close(3) = 0 [pid 8819] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3 [pid 8819] connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory) [pid 8819] close(3)

The first system call made in this stack trace is set_robust_list. This is not one that I've encountered before, so I headed over to the docs.

The set_robust_list() system call requests the kernel to record the head of the list of robust futexes owned by the calling thread.

Aha! Futexes. I learned about these a while ago in a systems class. Futexes are a low-level multi-threading construct. They allow you to implement locking (a way for a thread to claim control over a resource so that other threads cannot access it at the same time). You can read more about futexes here. I'm guessing that the curl process spins out some threads to do background work and this set_robust_list function call is related to those soon-to-be created threads.

Following that, the program attempts to open two socket connections. I wasn't sure what the point of these socket connections would be. It seemed too early to be opening any sockets in the program. I decided to do some Googling on the "/var/run/nscd/socket" bit to see if that might illuminate what the socket connections are for. I came across this blog post which explained that nscd is used to run a local DNS resolver. A local DNS resolver is a program that performs a lookup for an IP address for a particular domain on a local machine. As it turns out, the attempt to open a socket to the local DNS server on the machine failed because there isn't one running on the machine that I executed the curl command in. As it turns out, this relates to some system calls that are made later down the line.

Speaking of which, the next chunk of the stack trace looks like this.

[pid 8819] open("/etc/nsswitch.conf", O_RDONLY|O_CLOEXEC) = 3 [pid 8819] fstat(3, {st_mode=S_IFREG|0644, st_size=497, ...}) = 0 [pid 8819] read(3, "# /etc/nsswitch.conf\n#\n# Example"..., 4096) = 497 [pid 8819] read(3, "", 4096) = 0 [pid 8819] close(3) = 0

I'm not familiar with the nsswitch.conf file. That's no surprise, a lot of things in this stack trace are new to me. The docs revealed that this configuration file is used to store name resolution details for different. The important line in this file when it comes to curl is below.

hosts: files dns

This line states that host names should first be resolved by looking in any configuration files then through a DNS service.

The next chunk of the stack trace relates closely to what the nsswitch.conf file revealed.

[pid 8819] open("/etc/host.conf", O_RDONLY|O_CLOEXEC) = 3 [pid 8819] fstat(3, {st_mode=S_IFREG|0644, st_size=9, ...}) = 0 [pid 8819] read(3, "multi on\n", 4096) = 9 [pid 8819] read(3, "", 4096) = 0 [pid 8819] close(3)

In this case, curl looks through the host.conf, which is a file that dictates how hostnames are resolved to IP addresses. This file can have some verastile configurations but in this case it contains only one, multi on, which dictates that a host stored in the hosts file (/etc/hosts) can reference multiple IP addresses.

The next couple of lines in the stack trace relate to the other part of the configuration in nsswitch.conf. The curl command opens up the resolv.conf file which contains a list of name servers that our computer can query to resolve domain names.

[pid 8819] futex(0x7f21c8974a64, FUTEX_WAKE_PRIVATE, 2147483647) = 0 [pid 8819] getpid() = 8818 [pid 8819] open("/etc/resolv.conf", O_RDONLY|O_CLOEXEC) = 3 [pid 8819] fstat(3, {st_mode=S_IFREG|0644, st_size=111, ...}) = 0 [pid 8819] read(3, "domain c.sky-eniac-f13b2.internal"..., 4096) = 111 [pid 8819] read(3, "", 4096) = 0 [pid 8819] close(3)

So in summary, the first couple of system calls executed by the curl command show that the first task it handles is associated with resolving the IP address for the domain name that is entered in the command. In this case, it was https://safia.rocks. I think this point is a good place to end this blog post. I'll pick up looking through the rest of the stack trace in the next blog post. See (write?) you then!

[captainsafia]

Looking at how `curl` works through stack traces

It's the end of the week! Woohoo! I'm very excited for the weekend. It's the only time I have these days to get lots of coding done on Zarf.

The first couple of relevant lines in the stack trace are pretty self-explanatory. They capture writing the command to standard input (which is defined through file descriptor 2). The one call that intrigued me was the pselect6 call, primarily because I haven't encountered it before. As it turns out pselect6 is responsible for determining whether a file descriptor in a set of file descriptors is ready for interactions. The first two commands are the number of file descriptors to observe and the second is the set of file descriptors to observe. In this case, pselect is only observing the state of file descriptor 0. File descriptor 0 is the standard input file descriptor, so I'm assuming that this system call does something to the effect of checking to see if a user isn't actively typing a command.

write(2, "curl https://safia.rocks", 24) = 24 pselect6(1, [0], NULL, NULL, NULL, {[], 8}) = 1 (in [0]) read(0, "\r", 1) = 1 write(2, "\n", 1) = 1

The next couple of lines in the stack trace invoke a command that I'm familiar with using a rather interesting set of parameters. As mentioned in other blog posts, ioctl is a command that is

ioctl(0, TCGETS, {B38400 opost isig -icanon -echo ...}) = 0 ioctl(0, SNDCTL_TMR_STOP or TCSETSW, {B38400 opost isig icanon echo ...}) = 0 ioctl(0, TCGETS, {B38400 opost isig icanon echo ...}) = 0

The next few lines were variations of this. I've never encountered the rt_sigaction

rt_sigaction(SIGINT, {sa_handler=0x467410, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f 3750fe6060}, {sa_handler=0x4bb540, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f3750fe60 60}, 8) = 0

The next couple of lines in the stack trace made me realize that I had not configured strace properly before running my curl command. In the stack trace below, there is a call made to the clone syscall which creates a new process thread. Shortly after this new thread is created, the stack trace ends. This made me realize that a lot of the networking-related work must have been happening in that thread. In particular, I was looking to find system calls associated with opening sockets and recving messages from them.

pipe([3, 4]) = 0 clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0 x7f375199ae10) = 13439 = 0

I went back and reran the strace command with the appropriate flag, the f flag, which tells strace to follow child processes and threads. When I did this, I got a stack trace that was much longer and had a lot of the system calls that I expected to see. One of the first networking-related things that happen is related to configurations. In this case, it loads up the default configuration that a user might have set for curl.

[pid 13601] open("/usr/lib/ssl/openssl.cnf", O_RDONLY) = 3 ... [pid 13601] open("/home/safia/.curlrc", O_RDONLY)

As I scrolled through the rest of the stack trace, I realized that it was the kind of thing that should be analyzed in multiple series. Considering the amount of time I have to write these days, I have to keep my blog posts really short. I don't mind this too much. In this case, it'll allow me to parse out this rather hefty stack trace without losing it.

So stay tuned for that!

[captainsafia]

What happens when you run `sudo !!`?

One thing I'm sure everyone has done on the command line is to use the !! shortcut to run the command run previously with sudo.

$ gimme-the-secrets You can't run this command. $ sudo !! The answer to life, the universe, and everything is 42.

This blog post is gonna deviate a little bit from what I usually do because the logic for the !! shortcut doesn't utilize any system calls so looking at the stack trace wouldn't yield anything illuminating.

The !! command leverages the history file in which Bash maintains a record of the commands executed by the user. You can find the location of this file by running the following command.

$ echo $HISTFILE /home/captainsafia/.bash_history

I started off the investigation by running type on !.

$ type ! ! is a shell keyword

This makes sense. ! is a shell-level construct that is part of the shell's grammar. I suspect that the shell manages to replace any commands that reference "!" with the prior command executed. I'm actually still murky on this. I decided to hop over to the bash code base to see if I could dissect this a little bit.

I started by looking for references to "!" in the code base, but that didn't come up with anything because it probably would be referenced using a word, not a symbol (symbol in the normal sense, not the computer science one). I tried looking for "exclamation" and "keyword" but came up with nothing. Then I finally realized that it would probably be referenced using "bang" in the code base and got some useful snippets to look at from that.

It turns out that the ability to expand ! as a reference to an element in the history is governed by the -H flag that can be passed to a new bash instance and sets the BASH_HISTORY flag to true.

Some more snooping led me to the bashhist.c file in the source code which managers interactions with the history file. I started snooping around this file to see if I could find anything that was responsible for taking the "!" in a command and replacing it appropriately.

I found a promising result in the pre_process_line which handles expanding "!" into the most recent history entry. The function checks to see if history expansion is enabled in the current bash session and invokes the history_expand.

# if defined (BANG_HISTORY) /* History expand the line. If this results in no errors, then add that line to the history if ADDIT is non-zero. */ if (!history_expansion_inhibited && history_expansion && history_expansion_p (line)) { expanded = history_expand (line, &history_value);

I read through the code for the history_expand function and it essentially translates all the different forms of "!" that can be used, like "!!" to access the last command or "!2" to access the second command in the history file, into the actual commands that are referenced and returns them as a new string to be parsed by the shell.

So in summary, when you type "sudo !!" the following happens.

Bash begins pre-processing the line.

If history expansion is enabled, Bash passes the "!!" string to the history_expand function which returns the last item in the history file.

Bash begins pre-processing that new line. Since it has no history expansion strings in it, it just executes the commands referenced within.

Fin.

Of course, there's a lot more edge-casing checking and other business happening, but the above steps get to the meat of the process.

See you in the next post!

[captainsafia]

Looking at the stack trace for `ls`

You know what Mondays mean. A new blog post!

I'm trying something new this week. Instead of writing my thrice-weekly blog posts on my commute to and from work, I'll write all of them on a single day over the weekend.

Today, I'm going to be looking at a command that I previously looked at the codebase for, the ls command. To be honest, it feels like that happened so long ago I don't even remember it. In any case, I hopped over to my Debian VM instance on Google Cloud and ran strace on the ls command in my home directory.

It starts off by running the open system call on the current working directory. It's passing quite a few flags to the call. Some I already knew about and some I didn't. Nonetheless, here's a breakdown of what all those flags are doing.

O_RDONLY: Open the file for reading only. No file modifications allowed!

O_NONBLOCK: Return from the open call without any delay.

O_DIRECTORY: Open a directory with the intent of examining its contents.

O_CLOEXEC: Automatically close the file descriptor when the current process being execed returns.

[pid 13721] open(".", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3 [pid 13721] fstat(3, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0

The next system call invoked is the getdents system call which, according to the docs, returns the contents of a directory.

[pid 13721] getdents(3, /* 13 entries */, 32768) = 400

The next couple of lines are invocations of the lstat function on each of the contents of the directory. This is used by ls to determine the size of the file and the owner and so on. This is the kind of information that you would see when you run ls with the -l flag.

[pid 13721] lstat("test-4.txt", {st_mode=S_IFREG|0644, st_size=13, ...}) = 0

The last two calls in the stack trace did mystify me a bit. The function makes another call to the getdents system call. Why is it doing this? I tried to spot the differences between the two system calls. In the first one, the second parameter points to a list of 13 directory entry structs and the system call returns 400. In the second call listed below, the second parameter points to a list of 0 directory entry structs and returns 0. I figured that perhaps the invocation was made twice because there was some for-loop iterating until the getdents function returned 0.

[pid 13721] getdents(3, /* 0 entries */, 32768) = 0 [pid 13721] close(3)

The last line in the stack trace closes the file descriptor associated with the current directory that we are reading.

And with that, I close off this blog post! I know this blog post is short, but that's because I'm preparing you for what's coming later this week. It's gonna be a doozy...

[captainsafia]

What do `cp` and `mv` do under the hood?

Another Friday, another blog post! Can you believe I've written around 50 blog posts in the last four months? I certainly can't! I feel like it gets easier every time, so the adage is true: practice does make perfect.

Today, I'd like to make another helpless Unix command my victim. Well, two of them actually, I've recently been interested in the distinctions between the mv command and the copy command. I have the vaguest hunch that under the hood, they actually mostly do the same thing, but this blog isn't about unsubstantiated hunches. Let's see if we can see what the difference might be from the stack trace.

I ran strace on the following commands. mv test-1.txt test-2.txt and cp test-3.txt test-4.txt.

The stack trace for the mv command actually ended up being shorter and much easier to process. The first couple of lines are responsible for checking to see if a file with the filename that we want to mv to already exists.

I wasn't sure what the distinction between stat and lstat was. I knew what it was at some point but that was many moons ago, and I needed to jog my memory. It turns out that they both essentially do the same thing (return the attributes of an inode) expect that lstat doesn't run stat on the original inode pointed to by a symbolic link but on the symbolic link itself. It's a way of getting the stats on the reference instead of the actual file.

The last invocation is to the rename system call which is responsible for changing the name or location of a file (documentation) does most of the heavy lifting here.

stat("test-2.txt", 0x7ffd03574f90) = -1 ENOENT (No such file or directory) lstat("test-1.txt", {st_mode=S_IFREG|0644, st_size=13, ...}) = 0 lstat("test-2.txt", 0x7ffd03574c70) = -1 ENOENT (No such file or directory) rename("test-1.txt", "test-2.txt") = 0

The cp command had a lot more going on. As it turns out, they are quite different. I dunno why I thought they would be similar. In hindsight, that didn't make sense. I guess those are the kind of assumptions you make when you're writing blog posts at 50 miles an hour in a rideshare with a bunch of strangers without having had any coffee or tea yet.

The stack trace for the cp command starts with the same stat commands that were used in the mv command.

Then both files are opened, the existing file for reading and the new file for writing. I found the fstat calls that are referenced her to be interesting. Again, there is a murky recollection of what fstat does in the back of my mind. After looking up the documentation, I realized that looking at the parameters used in the system call would have actually revealed its purpose. The parameters, 3 and 4, are references to the file descriptors of the files that were opened for reading and writing. As such, fstat is used to fetch the file attributes of these files before writing to them.

There's a seemingly random call to the fadvise64 system call after that. I've never encountered this command, and after looking it up, I realized that it was pretty nifty. Apparently, it's how the program can inform the operating system of its intentions with respect to accessing a particular file. This way the operating system can make the appropriate optimizations depending on what those intentions are. For example, if the plan is to copy data, the operating system might position the data copied and the location copied to be closer to each other.

Afterward, data is read from the old file and written to the new file. There's not much fun stuff going on here, and I think the system calls are pretty self-explanatory.

stat("test-4.txt", 0x7ffd15bf1830) = -1 ENOENT (No such file or directory) stat("test-3.txt", {st_mode=S_IFREG|0644, st_size=13, ...}) = 0 stat("test-4.txt", 0x7ffd15bf15a0) = -1 ENOENT (No such file or directory) open("test-3.txt", O_RDONLY) = 3 fstat(3, {st_mode=S_IFREG|0644, st_size=13, ...}) = 0 open("test-4.txt", O_WRONLY|O_CREAT|O_EXCL, 0644) = 4 fstat(4, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 fadvise64(3, 0, 0, POSIX_FADV_SEQUENTIAL) = 0 mmap(NULL, 139264, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f875a6b600 0 read(3, "Lorem ipsum.\n", 131072) = 13 write(4, "Lorem ipsum.\n", 13) = 13 read(3, "", 131072) = 0 close(4) = 0 close(3) = 0

See you in the next blog post!

[captainsafia]

I finally figured out how Unix redirection works under the hood

In the last blog post, I decided to look into Unix redirection. Towards the end of it, I ended up being perplexed about an aspect of the stack trace and stated that I would look into it later.

Well, I lied.

Today, I'll be digging into the same concept using a different stack trace. As I mentioned in previous blog posts, since I started working full-time, I've been writing my thrice-weekly blog posts on my commute to work. Usually, I'll run dtruss on my Mac at home, copy the stack trace over, and then analyze it there. Recently, I was looking through some Google Cloud emails I was getting and realized that I had a one-year free trial of Google Cloud hanging around in my inbox. I used it to set up a Debian virtual machine that I could SSH to and use for these blog posts. So from now on, the stack traces I'm looking through are run on a Debian machine.

Sidebar: A lot of pedants on Hacker News like to point out that I've been reading stack traces on Mac, not a proper *nix system. I see where they're coming from, but I try to keep these blog posts to be as operating-system specific as possible. My analysis usually refers to system calls that are used across the *nix ecosystem for similar purposes. All this is to say, stop being pedantic and let a girl live. If you're genuinely concerned about what I'm writing, invest your own time in writing your own blog posts the way you want them written. Trust me; you'll be happier that way.

Anyway, angry sidebar aside, let's get back to the heart of the story. Before you read further, I'd recommend reading the last blog post. It contains some explanations of the fork and exec system calls and how the way processes are executed enables redirection.

Did you read it? OK. Onward we go!

So, like last time, I ran strace on a sample command to get the system calls invoked while it ran.

execve("/bin/echo", ["echo", "New addition."], [/* 16 vars */]) = 0 brk(NULL) = 0x55c3a89fa000 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory) mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f3a4faf9000 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3 fstat(3, {st_mode=S_IFREG|0644, st_size=13435, ...}) = 0 mmap(NULL, 13435, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f3a4faf5000 close(3) = 0 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory) open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3 read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\4\2\0\0\0\0\0"..., 832) = 832 fstat(3, {st_mode=S_IFREG|0755, st_size=1689360, ...}) = 0 mmap(NULL, 3795296, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f3a4f53a000 mprotect(0x7f3a4f6cf000, 2097152, PROT_NONE) = 0 mmap(0x7f3a4f8cf000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x195000) = 0x7f3a4f8cf000 mmap(0x7f3a4f8d5000, 14688, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f3a4f8d5000 close(3) = 0 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f3a4faf3000 arch_prctl(ARCH_SET_FS, 0x7f3a4faf3700) = 0 mprotect(0x7f3a4f8cf000, 16384, PROT_READ) = 0 mprotect(0x55c3a7464000, 4096, PROT_READ) = 0 mprotect(0x7f3a4fafc000, 4096, PROT_READ) = 0 munmap(0x7f3a4faf5000, 13435) = 0 brk(NULL) = 0x55c3a89fa000 brk(0x55c3a8a1b000) = 0x55c3a8a1b000 open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3 fstat(3, {st_mode=S_IFREG|0644, st_size=1679776, ...}) = 0 mmap(NULL, 1679776, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f3a4f958000 close(3) = 0 fstat(1, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 write(1, "New addition.\n", 14) = 14 close(1) = 0 close(2) = 0 exit_group(0) = ?

To decode this stack trace a little bit, I started by figuring out what happens from bottom to top.

The last system call, exit_group, is responsible for "closing all threads in the process." I wondered what this would mean in the context of the command currently being executed. In particular, I figured that if exit_group is responsible for closing all threads in a process then there should be some invocations to clone in the stack trace (clone creates a new thread). Eventually, I figured out the answer to why exit_group was being called although it seemed that no threads were created. According to this, in recent versions of Linux, the call to the exit function implicitly calls exit_group under the hood. It seems that this was done so that whenever you exited a process, it closed out the threads to instead of potentially leaving them open.

The two commands preceding it were pretty self-explanatory. close(1) and close(2) are responsible closing the file descriptors associated with the parameters passed. As mentioned in my last blog post, it looks like it is closing the file descriptors associated with standard input and output in that process.

The function call immediately preceding this actually prints out the string we are echoing into the file referenced by file descriptor 1. At this point, I'm pretty sure that somewhere in the stack trace I should see a function call to open file.txt but I don't see it. At this point, I'm suspecting that strace isn't calling the fork call that occurs before file output. I tried to run strace with the -f flag to follow the forks but didn't have any luck.

Eventually, I figured out what was going on. Similar to last time, to capture the system calls associated with the redirection, I had to run strace on the shell process that I was executing the commands in. Once I did this, I got the output that I expected.

open("file.txt", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3 fcntl(1, F_GETFD) = 0 fcntl(1, F_DUPFD, 10) = 10 fcntl(1, F_GETFD) = 0 fcntl(10, F_SETFD, FD_CLOEXEC) = 0 dup2(3, 1) = 1 close(3) = 0 write(1, "New addition.\n", 14) = 14 dup2(10, 1) = 1 fcntl(10, F_GETFD) = 0x1 (flags FD_CLOEXEC) close(10)

That's more like it! In this stack trace, I can see the open system call that opens the file.txt we are writing too. This seems to be the stack trace I actually want to read.

I'll go through this one from top to bottom. The first line in the stack trace opens the file.txt file for writing. The return code, 3, tells us the file descriptor associated with this open file.

The next four system calls all invoke the fcntl command which manipulates file descriptors based on its second parameter. The first call to fnctl gets the flags associated with the standard output file descriptor. The second function duplicates the file referenced by file descriptor 1 into file descriptor 10. The next two system calls are responsible for setting the flags on the new file descriptor to match the old one. So essentially, all this code is doing is creating a copy of the file descriptor associated with standard output.

The next system call, dup2, is one I have not encountered before. Looking at the documentation, it looks like this function copies the file descriptor 3 into the file descriptor 1. This effectively remaps standard output to file.txt.

After that, we write the text we want directly to file descriptor 1, which is also file.txt.

The last few function calls undo the remapping of the standard output file descriptor and close out any stale file descriptors.

This makes much more sense! Essentially, we make a backup copy of the file descriptor typically associated with standard output and store it in a new file descriptor. Then we map the file descriptor associated with standard output to our file. Then after writing, we undo the re-mappings we made.

I guess I had not been properly calling the strace command earlier which is why I was getting all that funky output. Once I figured out the way to properly call strace in order to process redirection correctly, I got a much clearer stack trace!

Huzzah!

[captainsafia]

Reveling in redirects: exploring Unix input/output redirection

One of my favorite Linux features is redirection. Redirection gives you the ability to send the output of one command directly to another. For example, here's how I would copy the contents of a file into my computer's clipboard using a pipe redirection.

$ cat file.txt | pbcopy

I can also use redirection to add some text to a file.

$ echo "New stuff." > file.txt

All of this got me curious about something: how does redirection work? I have a partial answer to this question from my university coursework on computer science. It relates to the way that Unix executes a process. In this case, two system calls will be used: fork and exec.

fork is a command that copies all the details of the currently running process. A process is an entity that represents a program that is currently running. In Linux, the details of a process are stored in task_struct You can read through the definition of the task_struct in this file in the Linux code base. When a process is forked, a copy of the task_struct associated with that process is made. If you read through the struct definition, you'll notice it stores things like the state of the process or the pidof the process.

exec is a command that overwrites the attributes of the currently running process with the details associated with a program (like the ls or cat programs). Generally what happens is that the fork system call is made a new copy of the currently running process is made. This copy is now the new currently running process. Then the exec call is made, and a new program is loaded into that process.

The fact that the act of creating a new process and starting to execute a program are distinct steps allow us to do some interesting things in between the fork and exec steps. For example, implementing a redirect from an echo command to a file as seen in the example above would look like.

Open the file file.txt with a file descriptor of 1.

Fork the currently running process.

Execute the command echo "New stuff.

They key here is the fact that we opened the file file.txt with a file descriptor set to 1. File descriptors are an abstract way of representing how any input/output resource (like a file) should be accessed. Generally, all process have access to three file descriptors numbered 0, 1, and 2. These numbers seem arbitrary (and that's kinda because they are). A better way to refer to them is through the constants that they are assigned to which are STDIN_FILENO, STDOUT_FILENO, and STDERR_FILENO, respectively.

So in essence, in the steps above, we are mapping over the file descriptor associated with standard out to a file (instead of output on the console). When we fork the process, this file descriptor mapping is copied over as well and used when we execute the echo command.

OK. So now that I've explained all that, I want to try to actually see it in action. To do this, I hopped on over to the good ol' dtruss tool and tried to figure out what a stack trace of the command echo "New text." > file.txt might be able to tell me about this.

Sidebar: Since I was trying to find the stack trace for a command that included redirection, I couldn't just run dtruss <command here>as usual. Instead, I ended up connecting dtruss to a bash session and looking through the stack trace associated with the commands executed in that session. Below, I've extracted the system calls that I think are associated with redirection.

29949/0x6d9b91: fork() = 0 0 29949/0x6d9b91: thread_selfid(0x0, 0x0, 0x0) = 7183249 0 29949/0x6d9b91: issetugid(0x0, 0x0, 0x0) = 0 0 29949/0x6d9b91: csrctl(0x0, 0x7FFEE483CF7C, 0x4) = -1 Err#1 29949/0x6d9b91: csops(0x0, 0x0, 0x7FFEE483D890) = 0 0 29949/0x6d9b91: shared_region_check_np(0x7FFEE483CDD8, 0x0, 0x0) = 0 0 29949/0x6d9b91: stat64("/private/var/db/dyld/dyld_shared_cache_x86_64h\0", 0x7FFEE483CD20, 0x0) = 0 0 29949/0x6d9b91: csrctl(0x0, 0x7FFEE483CADC, 0x4) = -1 Err#1 29949/0x6d9b91: getpid(0x0, 0x0, 0x0) = 29949 0 29949/0x6d9b91: proc_info(0x2, 0x74FD, 0x16) = 1272 0 29949/0x6d9b91: ioctl(0x3, 0x80086804, 0x7FFEE483CE50) = 0 0 29949/0x6d9b91: close(0x3) = 0 0 29949/0x6d9b91: access("/AppleInternal/XBS/.isChrooted\0", 0x0, 0x0) = -1 Err#2 29949/0x6d9b91: thread_selfid(0x0, 0x0, 0x0) = 7183249 0 29949/0x6d9b91: bsdthread_register(0x7FFF56790BEC, 0x7FFF56790BDC, 0x2000) = 1073742047 0 29949/0x6d9b91: issetugid(0x0, 0x0, 0x0) = 0 0 29949/0x6d9b91: mprotect(0x10B3CB000, 0x1000, 0x0) = 0 0 29949/0x6d9b91: mprotect(0x10B3D0000, 0x1000, 0x0) = 0 0 29949/0x6d9b91: mprotect(0x10B3D1000, 0x1000, 0x0) = 0 0 29949/0x6d9b91: mprotect(0x10B3D6000, 0x1000, 0x0) = 0 0 29949/0x6d9b91: mprotect(0x10B3C9000, 0x88, 0x1) = 0 0 29949/0x6d9b91: mprotect(0x10B3D7000, 0x1000, 0x1) = 0 0 29949/0x6d9b91: mprotect(0x10B3C9000, 0x88, 0x3) = 0 0 29949/0x6d9b91: mprotect(0x10B3C9000, 0x88, 0x1) = 0 0 29949/0x6d9b91: getpid(0x0, 0x0, 0x0) = 29949 0 29949/0x6d9b91: stat64("/AppleInternal/XBS/.isChrooted\0", 0x7FFEE483C528, 0x0) = -1 Err#2 29949/0x6d9b91: stat64("/AppleInternal\0", 0x7FFEE483C5C0, 0x0) = -1 Err#2 29949/0x6d9b91: csops(0x74FD, 0x7, 0x7FFEE483C060) = 0 0 dtrace: error on enabled probe ID 2190 (ID 557: syscall::sysctl:return): invalid kernel access in action #11 at DIF offset 28 29949/0x6d9b91: csops(0x74FD, 0x7, 0x7FFEE483B950) = 0 0 29949/0x6d9b91: writev(0x1, 0x7FE5BE601210, 0x2) = 13 0

So the first system call made is the fork system call that I discussed earlier. Based on my knowledge, the next few system calls after it should be associated with mapping the standard output file descriptor to the file.txt file. I tried to pick out the system calls that might be associated with this and isolated the following.

29949/0x6d9b91: ioctl(0x3, 0x80086804, 0x7FFEE483CE50) = 0 0 29949/0x6d9b91: close(0x3) = 0 0

ioctlas discussed in previous blog posts is the system call associated with managing interactions it I/O resources like files. The first parameter passed to this function is the associated file descriptor. In this case, it is 0x3. I figure that this is the call associated with opening the file.txt file because the first 0-2 file descriptors are used by the operating system as mentioned above.

The other significant function call is all the way at the bottom.

29949/0x6d9b91: writev(0x1, 0x7FE5BE601210, 0x2) = 13 0

I've never encountered this writev system call before, so I decided to research it further. When I read the documentation, the most approachable definition I found for the system call was as follows.

The writev() system call works just like write(2) except that multiple buffers are written out.

This is my hunch, but I'm perplexed by the fact that the file descriptors don't match up. If we open the file at file descriptor 0x3 and write to file descriptor 0x1 then where is the actual redirection happening? I wondered if the request code passed to the ioctl did something to connect the two file descriptor. Or maybe there was some code that took care of executing the redirection that wouldn't show up in the system call?

In any case, this blog post is getting a little long, and my commute to work is almost over (I write these blog posts on the trip to work), so I'll have to continue this particular investigation in another blog post.

See you then! And hopefully, I'll have more clarity on this whole affair at that point.

[captainsafia]

What happens when you run `cp` on the command line?

It's time for another edition of whatever the heck this is! After the deep dives I took into the Git code base, I no longer have the energy to craft up interesting names for these code anthropology moments.

I was considering diving into how the cp command works in this blog post. If you're a long-time reader, you might know that I generally do my code reads on my Mac. Since I started my full-time job on Monday, I've been writing these blog posts on a Chromebook on the way to and from work (amongst a variety of other things I do on my commute like stare at traffic and acknowledge how wonderful it is that will hopefully never have to operate a motor vehicle). Since I'm working on my Chromebook, I'll usually run the dtruss command on my Mac before I head out the door, copy and paste the output of the stack trace onto cloud-enabled text editor, then read through and research parts of the stack trace on my way to work.

Sidebar: I think at some point I'm going to irritate somebody on this rideshare with all this clackity-clacking!

This is all to say that these blog posts might be shorter and jumpier than usual due to the new constraints that I am writing under. Let's call it eXtreme Writing. Heh. Get it? Like eXtreme Programming? OK. I'll keep my day job.

With all that said, it's time to get to the real purpose of this post. What happens when you run the cp command? The cp command is often used to make copies of directories and files. But what is going on under the hood?

$ sudo dtruss cp file.txt file-2.txt SYSCALL(args) = return open("/dev/dtracehelper\0", 0x2, 0xFFFFFFFFE1B5FB00) = 3 0 ioctl(0x3, 0x80086804, 0x7FFEE1B5FA60) = 0 0 close(0x3) = 0 0 access("/AppleInternal/XBS/.isChrooted\0", 0x0, 0x0) = -1 Err#2 thread_selfid(0x0, 0x0, 0x0) = 6158775 0 bsdthread_register(0x7FFF56790BEC, 0x7FFF56790BDC, 0x2000) = 1073742047 0 issetugid(0x0, 0x0, 0x0) = 0 0 mprotect(0x10E1AB000, 0x1000, 0x0) = 0 0 mprotect(0x10E1B0000, 0x1000, 0x0) = 0 0 mprotect(0x10E1B1000, 0x1000, 0x0) = 0 0 mprotect(0x10E1B6000, 0x1000, 0x0) = 0 0 mprotect(0x10E1A9000, 0x88, 0x1) = 0 0 mprotect(0x10E1B7000, 0x1000, 0x1) = 0 0 mprotect(0x10E1A9000, 0x88, 0x3) = 0 0 mprotect(0x10E1A9000, 0x88, 0x1) = 0 0 getpid(0x0, 0x0, 0x0) = 8855 0 stat64("/AppleInternal/XBS/.isChrooted\0", 0x7FFEE1B5F138, 0x0) = -1 Err#2 stat64("/AppleInternal\0", 0x7FFEE1B5F1D0, 0x0) = -1 Err#2 csops(0x2297, 0x7, 0x7FFEE1B5EC70) = 0 0 dtrace: error on enabled probe ID 2190 (ID 557: syscall::sysctl:return): invalid kernel access in action #10 at DIF offset 28 csops(0x2297, 0x7, 0x7FFEE1B5E560) = 0 0 sigaction(0x1D, 0x7FFEE1B602E8, 0x7FFEE1B60310) = 0 0 stat64("file-2.txt\0", 0x7FFEE1B607D8, 0x0) = -1 Err#2 lstat64("file.txt\0", 0x7FFEE1B60868, 0x0) = 0 0 umask(0x1FF, 0x0, 0x0) = 18 0 umask(0x12, 0x0, 0x0) = 511 0 fstatat64(0xFFFFFFFFFFFFFFFE, 0x7FFB66D001C8, 0x7FFB66D001D8) = 0 0 stat64("file-2.txt\0", 0x7FFEE1B608F8, 0x0) = -1 Err#2 open("file.txt\0", 0x0, 0x0) = 3 0 open("file-2.txt\0", 0x601, 0x81A4) = 4 0 fstatfs64(0x4, 0x7FFEE1B5FA88, 0x0) = 0 0 fstat64(0x4, 0x7FFEE1B5FA88, 0x0) = 0 0 fchmod(0x4, 0x8180, 0x0) = 0 0 dtrace: error on enabled probe ID 2166 (ID 159: syscall::read:return): invalid kernel access in action #12 at DIF offset 68 dtrace: error on enabled probe ID 2164 (ID 161: syscall::write:return): invalid kernel access in action #12 at DIF offset 68 dtrace: error on enabled probe ID 2166 (ID 159: syscall::read:return): invalid kernel access in action #12 at DIF offset 68 fchmod(0x4, 0x81A4, 0x0) = 0 0 fstat64_extended(0x3, 0x7FFB66D00288, 0x7FFB66D003C0) = 0 0 fstat64(0x4, 0x7FFEE1B5F950, 0x0) = 0 0 fchmod(0x4, 0x1A4, 0x0) = 0 0 __mac_syscall(0x7FFF564ACD02, 0x52, 0x7FFEE1B5F8D0) = -1 Err#93 flistxattr(0x4, 0x0, 0x0) = 0 0 flistxattr(0x3, 0x0, 0x0) = 0 0 fchmod(0x4, 0x1A4, 0x0) = 0 0 close(0x3) = 0 0 close(0x4) = 0 0

So first things first, I've been doing this stack trace reads for a while and I'm rather curious about this bit.

syscall::read:return): invalid kernel access in action #12 at DIF offset 68

It seems like this might be related to a process attempting to access memory blocks that it does not have access to. I decided to Google around to see what the Internet might have to say about this. After looking through several forum posts, Apple support posts, blog posts, and stack exchange posts, it appears that this error is related to Apple's system integrity protection. It's a feature that was released in El Captain that restricts what access the root user has and what they can do on protected parts of the operating system. You can read more about it here. It's actually a pretty neat concept.

Admittedly, I had a pretty easy time scrolling through this stack trace and figuring out what was going on. Most of the commands used are ones that I'm already familiar with, and it was generally easy to map how the program would flow.

There were two parts that caught my eye. This:

__mac_syscall(0x7FFF564ACD02, 0x52, 0x7FFEE1B5F8D0) = -1 Err#93

And this:

flistxattr(0x4, 0x0, 0x0) = 0 0 flistxattr(0x3, 0x0, 0x0) = 0 0

I figured it would be easy to start off by looking into that second syscall, since the __mac_syscall bit is completely opaque to me. I started to look into the flistxattr command and by "look into" I mean "Google." I found some useful documentation on that here. So it looks like this command is adding a new set of attributes representing each of the files. The function declaration of this function looks like this.

ssize_t flistxattr(int fd, char *list, size_t size);

So from the trace above, it looks like the pare parameter that is different between the two syscalls is fd, which is a reference to the file descriptor. In this case, I assume it is setting the name:value attributes on the original file and the copy.

Now to that interesting __mac_syscall(0x7FFF564ACD02, 0x52, 0x7FFEE1B5F8D0) bit. What I found was interesting was that the result of this function call resulted in an error. To investigate what this call does, I did some Googling. After reading a blog post, several StackOverflow posts, and a thesis, I eventually found the implementation for this function in the Darwin code base. You can read it here. The most important part of the code snippet I found is the comment on it.

/* * __mac_syscall: Perform a MAC policy system call * * Parameters: p Process calling this routine * uap User argument descriptor (see below) * retv (Unused) * * Indirect: uap->policy Name of target MAC policy * uap->call MAC policy-specific system call to perform * uap->arg MAC policy-specific system call arguments * * Returns: 0 Success * !0 Not success * */ int __mac_syscall(proc_t p, struct __mac_syscall_args *uap, int *retv __unused)

I did some research to figure out what MAC policies were and discovered that they are a security-related construct. You can read more about them here. After reading that referenced link, that system call above makes a whole lot more sense. With that little bit squared away, the stack trace has a bit more clarity too.

Until the next post!

[captainsafia]

Peeking into `pwd`

Oh my gosh. Can I really keep up with this? I'm gonna be honest with you, fair reader, working a full-time job and maintaining this blog is proving to be quite the challenge.

But alas, I shall persist.

I figured I might continue the trend that I started in my last blog post and dive into the layers beneath some common command line tools. Today, I'm interested in one that is extremely common: pwd.

$ pwd /Users/captainsafia

As I did in the last blog post, I'll start diving into it by using the dtrace tool that comes with macOS. It's essentially a command line utility that allows you to analyze a trace of the system calls made during the execution of a program. Here's what the dtrace program outputs when I run pwd on my home directory.

sudo dtruss /tmp/pwd dtrace: system integrity protection is on, some features will not be available SYSCALL(args) = return /Users/captainsafia open("/dev/dtracehelper\0", 0x2, 0xFFFFFFFFE2CBAB20) = 3 0 ioctl(0x3, 0x80086804, 0x7FFEE2CBAA80) = 0 0 close(0x3) = 0 0 access("/AppleInternal/XBS/.isChrooted\0", 0x0, 0x0) = -1 Err#2 thread_selfid(0x0, 0x0, 0x0) = 6082815 0 bsdthread_register(0x7FFF56790BEC, 0x7FFF56790BDC, 0x2000) = 1073742047 0 issetugid(0x0, 0x0, 0x0) = 0 0 mprotect(0x10CF4D000, 0x1000, 0x0) = 0 0 mprotect(0x10CF52000, 0x1000, 0x0) = 0 0 mprotect(0x10CF53000, 0x1000, 0x0) = 0 0 mprotect(0x10CF58000, 0x1000, 0x0) = 0 0 mprotect(0x10CF4B000, 0x88, 0x1) = 0 0 mprotect(0x10CF59000, 0x1000, 0x1) = 0 0 mprotect(0x10CF4B000, 0x88, 0x3) = 0 0 mprotect(0x10CF4B000, 0x88, 0x1) = 0 0 getpid(0x0, 0x0, 0x0) = 8043 0 stat64("/AppleInternal/XBS/.isChrooted\0", 0x7FFEE2CBA158, 0x0) = -1 Err#2 stat64("/AppleInternal\0", 0x7FFEE2CBA1F0, 0x0) = -1 Err#2 csops(0x1F6B, 0x7, 0x7FFEE2CB9C90) = 0 0 dtrace: error on enabled probe ID 2190 (ID 557: syscall::sysctl:return): invalid kernel access in action #10 at DIF offset 28 csops(0x1F6B, 0x7, 0x7FFEE2CB9580) = 0 0 stat64("/Users/captainsafia\0", 0x7FFEE2CBB8B0, 0x0) = 0 0 stat64(".\0", 0x7FFEE2CBB940, 0x0) = 0 0 getrlimit(0x1008, 0x7FFEE2CBB6D0, 0x0) = 0 0 fstat64(0x1, 0x7FFEE2CBB6E8, 0x0) = 0 0 ioctl(0x1, 0x4004667A, 0x7FFEE2CBB734) = 0 0 dtrace: error on enabled probe ID 2165 (ID 947: syscall::write_nocancel:return): invalid kernel access in action #12 at DIF offset 68

Oh boy! There's quite a lot going on here. But we can rule most of it out since it was explored in the last blog post. There were a few lines in the stack trace that attracted my interest. The first line was this one.

getrlimit(0x1008, 0x7FFEE2CBB6D0, 0x0) = 0 0

What does getrlimit do? I did some digging on the Internet and discovered that is responsible for getting resource limits on a resource. In this case, the first parameter (0x1008) is a reference to the resource we are trying to modify limits on. I wondered what resource this number was referencing. It would go a long way towards helping me figure out why this function was invoked in this bit of code. I ended up finding something useful in the Python documentation. Python has a built-in resource module that can be used to interface with resources on the system. There was this interesting quote in the documentation.

The Unix man page for getrlimit(2) lists the available resources. Note that not all systems use the same symbol or same value to denote the same resource. This module does not attempt to mask platform differences — symbols not defined for a platform will not be available from this module on that platform.

So it looks like each operating system has different values that it uses for that first parameter. This makes it difficult to figure out what exactly 0x1008 means in the context of macOS since a lot of the documentation is around Unix. That being said, the documentation did give me a pretty good idea of what the resource might reference. A resource is something like the number of bytes available to a process in its heap or the maximum amount of CPU time that the process can use.

Another thing I was curious about was this stat64 system call.

stat64(".\0", 0x7FFEE2CBB940, 0x0) = 0 0

I think this line is the most significant one concerning analyzing how pwd works because the parameter passed to it is the "." parameter, which is generally used as a reference to the current working directory.

Sidebar: The \0 is the null terminator and signifies that the string has ended.

As mentioned in the previous blog post, most of the other system calls are related to the actual operations of the dtrace tool and don't relate to the functionality of pwd.

That was an interesting exploration! The most fascinating bit was some of the snooping that I did around resource limits in the operating system. It exposed me to yet another way that operating systems are different at the low-level: the use different reference codes for resources. Of course, this small difference makes it difficult to build cross-platform applications using low-level programming langauges.

Until next time!

[captainsafia]

Unraveling `rm`: what happens when you run it?

Another Monday, another blog post!

I've been diving into the curl codebase over the past couple of blog posts, but something else has spiked my interest today, so I figured I might as well dig into it while the curiosity is still hot.

To be honest, I'm reluctant to dive into this because the last time I wrote a blog post about a Unix-related topic the — let's call them the group of individuals with too much time on their hands and a lot of petty on their hearts — took me to task on some of the substance in the blog post in a way that wasn't too nice. And by "wasn't too nice" I mean hella racist and sexist. In any case, I figure there will always be haters (and people with unhealthy attachments to operating systems and harassing strangers on the Internet), so I might as well carry on.

OK. Enough blabber. I've been working through a backlog of issues on the Zarf app. As such, I've been spending a lot of time on the command line. The backlog involved deleting a lot of code (insert satisfied sigh here) and sometimes this involved deleting entire files of source code (insert doubly satisfied sigh here). This got me wondering: what's going on when you run rm on the command line. There's a couple of variants of the rm command that I commonly run.

$ rm settings.json $ rm -rf config/

Anyways, I wanted to dive into what is going on under the hood with rm, so I decided to start by determining the syscalls invoked by the rm command.

captainsafia@eniac ~/zarf> sudo dtruss /tmp/rm History.md dtrace: system integrity protection is on, some features will not be available SYSCALL(args) = return open("/dev/dtracehelper\0", 0x2, 0xFFFFFFFFE9A3EB10) = 3 0 ioctl(0x3, 0x80086804, 0x7FFEE9A3EA70) = 0 0 close(0x3) = 0 0 access("/AppleInternal/XBS/.isChrooted\0", 0x0, 0x0) = -1 Err#2 thread_selfid(0x0, 0x0, 0x0) = 3765033 0 bsdthread_register(0x7FFF56790BEC, 0x7FFF56790BDC, 0x2000) = 1073742047 0 issetugid(0x0, 0x0, 0x0) = 0 0 mprotect(0x1061CA000, 0x1000, 0x0) = 0 0 mprotect(0x1061CF000, 0x1000, 0x0) = 0 0 mprotect(0x1061D0000, 0x1000, 0x0) = 0 0 mprotect(0x1061D5000, 0x1000, 0x0) = 0 0 mprotect(0x1061C8000, 0x88, 0x1) = 0 0 mprotect(0x1061D6000, 0x1000, 0x1) = 0 0 mprotect(0x1061C8000, 0x88, 0x3) = 0 0 mprotect(0x1061C8000, 0x88, 0x1) = 0 0 getpid(0x0, 0x0, 0x0) = 77384 0 stat64("/AppleInternal/XBS/.isChrooted\0", 0x7FFEE9A3E148, 0x0) = -1 Err#2 stat64("/AppleInternal\0", 0x7FFEE9A3E1E0, 0x0) = -1 Err#2 csops(0x12E48, 0x7, 0x7FFEE9A3DC80) = 0 0 dtrace: error on enabled probe ID 2190 (ID 557: syscall::sysctl:return): invalid kernel access in action #10 at DIF offset 28 csops(0x12E48, 0x7, 0x7FFEE9A3D570) = 0 0 geteuid(0x0, 0x0, 0x0) = 0 0 ioctl(0x0, 0x4004667A, 0x7FFEE9A3F954) = 0 0 lstat64("History.md\0", 0x7FFEE9A3F8F8, 0x0) = 0 0 access("History.md\0", 0x2, 0x0) = 0 0 unlink("History.md\0", 0x0, 0x0) = 0 0

Sidebar: Usually, you determine the syscalls utilized by a command by using strace. I'm on a Mac so strace isn't available. Instead, I used a tool called dtrace. To allow it to process the rm command, I had to make a copy of the executable into a temporary directory and execute that. All this to say, this is why I'm executing dtrace /tmp/rm above instead of strace rm.

So, anyway, let's look into what's going on above. The first couple of lines in the trace seem to be pretty clearly related to setting up the sudo part of the command. I was intrigued by the calls to the mprotect command. I figured that it might be something related to memory addresses because the first parameter passed to the mprotect function looks like a memory address. I decided to head over to Google to see if I could find the documentation for this function and confirm this. I find the documentation here and my suspicions were confirmed. The function is responsible for setting the access rights on memory for the calling process. The function declaration looks like this int mprotect(void *addr, size_t len, int prot) where addr is the start of the memory range, and len is the length of range of memory addresses that will be changed, and prot is an integer that represents how the memory should be protected. I looked into what the different values for prot that were passed into the mprotect function calls above and figured out the following.

0x0 is the code for PROT_NONE meaning that the memory cannot be accessed for writes or reads.

0x1 is the code for PROT_READ meaning that the memory can be read.

0x3 is a bitwise OR of the values for PROT_READ and PROT_WRITE which means that it allows that memory to be both read or written.

I wasn't sure what the memory addresses that were referenced in the mprotect call actually corresponded to or what the best way to figure it would be.

The getpid command has a pretty self-explanatory name, but I wondered what the parameters that were passed to the function were. As it turns out, the getpid function supposedly takes no parameters, so the inclusion of the parameters in the call above perplexed me.

I was also unsure of what the csops and ioctl calls did. As it turns out, this is actually pretty warranted. Some investigation revealed that csops is a system call that is unique to the Apple operating system and can be used to check the signature that is written into a memory page by the operating system. It seems to be some way of checking the validity of a particular chunk of code. I'm not too sure about it, and there isn't a ton of information about Apple-specific syscalls so I can't dig into the details of this as well as I'd like.

The ioctl syscall is a pretty versatile one and is responsible for all input/output related interactions. According to the manpage, the first argument represents a file descriptor, the second is a reference to a "device-dependent request code", and the third is a pointer to memory. I dug around to figure out what the request code "0x4004667A" and realized that it was pretty commonly associated with invocations of the dtrace command so I figured that this invocation was not related to the task of removing the file referenced.

I was interested in the last syscall invoked in this strace, unlink. I headed back to Google to learn some more about it and came across this manpage. The unlink command is the one that is actually responsible for removing the file.

All in all, the most rm-related parts of the trace are in the last few lines. Everything else seems to be setup associated with setting up memory permissions, ensuring the status of the related files, and setting up the dtrace command.

So there's that! I'll admit that I'm not sure how I feel about this trace approach to antropology. It's a little too direct, I almost like wallowing in the code and getting last in the complexity of it. There is something therapeutic about it. I'll see if I get more comfortable with this technique in future code reads. Until then, see you next time!

[captainsafia]

Digging further into the curl code base

In the last blog post I wrote, I learned how curl maintains the configuration details for different operations. In this blog post, I'd like to figure out how curl executes these operations. Specifically, I'd like to dive into what is going on in the segments of code outlined below.

/* Perform each operation */ while(!result && config->current) { result = operate_do(config, config->current); config->current = config->current->next; if(config->current && config->current->easy) curl_easy_reset(config->current->easy); }

So what's going on here? As mentioned previously, the configuration for each of the operations that curl is executing is stored in a doubly linked list.

Sidebar: For those who are not familiar with the doubly linked list data structure, it is similar to a linked list, but in addition to having a pointer to the next node in each node, you also have a pointer to the previous point. Read this for more.

The while loop above iterates through each of the configurations that are currently stored in memory and executes the correlated operation. The most interesting (and ominous sounding) function here is the operate_do function. You can tell by its name, but the operate_do function sure does a whole lot. I kid you not; the entire function is around 1,700 lines. Is that even legal to do? I feel like there should be a rule against doing something like this. How do you even begin to investigate a function that long?

I'll tell ya how. You scroll through the code really fast. I'm not kidding. When a function is this long, you can often see the patterns visually. For example, as I scrolled through the code, I realized that a fair chunk of the code contained invocations to functions like my_setopt_str and my_setopt and my_setopt_enum. If I had to eyeball it, I would say that about 70% of the lines of code in the function were invocations to my_setopt* style functions.

These functions take as input a curl parameter which is a reference to a CURL object stored in the configuration and the configuration item that should be set up. I wasn't too interested in digging into this function. Looking at the function name and parameters gave me a rough idea of the purpose of these chunks of code: "before we actually execute an operation, make sure that its operation configurations are stored in the correct place."

Once I figured out the my_setopt trend in the operate_do function, I tried to look through the noise and see if I could find where the actual "doing" was happening. There were a couple of interesting chunks. For example, the code below is an introduction to the logic that iterates through each of the URLs in the URL list and executes a particular set of functions over each URL.

/* ** Nested loops start here. */ /* loop through the list of given URLs */ for(urlnode = config->url_list; urlnode; urlnode = urlnode->next) {

For example, it executes code associated with uploading files to a particular API endpoint.

/* Here's the loop for uploading multiple files within the same single globbed string. If no upload, we enter the loop once anyway. */ for(up = 0 ; up < infilenum; up++) { ... }

And most of the my_setopt functions that I observed above were located within the for-loop iterating through each of the URLs.

The last thing that I wanted to investigate was where the result object that is returned from the operate_do function is manipulated. This object is an integer status code that represents the status of the curl execution (whether it has run into an error or if a parameter is missing). Since it is a representation of a particularly important status code, anytime it is modified it is probably associated with an important function call. Here are all the places where the result object is modified in the operate_do function.

result = CURLE_FAILED_INIT; (L241)

result = curl_easy_getinfo(config->easy, CURLINFO_TLS_SSL_PTR, &tls_backend_info);(L237-L239)

Woah. I've only written up to of the references, and I've realized that if I do them all, I'm gonna be here all day and I've got a lot of errands to run. I'll spare you the pain of having to read through all the references and summarize the trend that I am seeing. Essentially, the result is modified whenever an HTTP request fails or if some sort of SSL certificate is configured improperly or if there isn't enough memory to write the output of a request and so on and so on. Reading through the places in the code where the result was modified gave me a lot of perspective into how much heavy lifting the operate_do function did, aka all of it.

I'm debating whether I want to dive more rigorously into this function or not. Part of me feels like there is more to be discovered but another part of me feels like I need to get more organized about my approach for reading through this code. After thinking about it for a little bit, I decided that I would try to do another code dive by observing what happens specifically when the following curl command is executed.

$ curl https://safia.rocks

Until next time!