My own perf comparison: when I switched from Fil-C running on my system’s libc (recent glibc) for yololand to my own build of musl, I got a 1-2% perf regression. My best guess is that it’s because glibc’s memcpy/memmove/memset are better. Couldn’t have been the allocator since Fil-C’s runtime has its own allocator.
- Userland: the place where you C code lives. Like the normal userland you're familiar with, but everything is compiled with Fil-C, so it's memory safe.
- Yololand: the place where Fil-C's runtime lives. Fil-C's runtime is about 100,000 lines of C code (almost entirely written by me), which currently has libc as a dependency (because the runtime makes syscalls using the normal C functions for syscalls rather than using assembly directly; also the runtime relies on a handful of libc utility functions that aren't syscalls, like memcpy).
So Fil-C has two libc's. The yololand libc (compiled with a normal C compiler, only there to support the runtime) and the userland libc (compiled with the Fil-C compiler like everything else in Fil-C userland, and this is what your C code calls into).
skissane 20 hours ago [-]
Why does yoyoland need to use libc’s memcpy? Can’t you just use __builtin_memcpy?
On Linux, if all you need is syscalls, you can just write your own syscall wrapper-like Go does.
Doesn’t work on some other operating systems (e.g. Solaris/Illumos, OpenBSD, macOS, Windows) where the system call interface is private to the system shared libraries
pizlonator 18 hours ago [-]
> Why does yoyoland need to use libc’s memcpy? Can’t you just use __builtin_memcpy?
Unless you do special things, the compiler turns __builtin_memcpy into a call to memcpy. :-)
There is __builtin_memcpy_inline, but then you're at the compiler's whims. I don't think I want that.
A faithful implementation of what you're proposing would have the Fil-C runtime provide a memcpy function so that whenever the compiler wants to call memcpy, it will call that function.
> On Linux, if all you need is syscalls, you can just write your own syscall wrapper-like Go does.
I could do that. I just don't, right now.
You're totally right that I could remove the yolo libc. This is one of like 1,000 reasons why Fil-C is slower than it needs to be right now. It's a young project so it has lots of this kind of "expedient engineering".
imcritic 10 hours ago [-]
You keep repeating the name wrong: yololand, not yololand.
cryptonector 4 hours ago [-]
GP was saying 'yoyoland', when it's 'yololand' (as in YOLO?).
pizlonator 3 hours ago [-]
Yeah YOLO.
I needed a fun term to refer to the C that isn’t Fil-C. I call it Yolo-C.
Hence yololand - the part of the Fil-C process that contains a bit of Yolo-C code for the Fil-C runtime.
cryptonector 2 hours ago [-]
Thanks. I went looking and saw this in the Fil-C manifesto:
> It's even possible to allocate memory using malloc from within a signal handler (which is necessary because Fil-C heap-allocates stack allocations).
Hmm, really? All stack allocations are heap-allocated? Doesn't that make Fil-C super slow? Is there no way to do stack allocation? Or did I misread what you meant by 'stack allocations'?
pizlonator 2 hours ago [-]
It’s a GC allocation, not a traditional malloc allocation. So slower than stack allocation but substantially faster than a malloc call.
And that GC allocation only happens if the compiler can’t prove that it’s nonescaping. The overwhelming majority of what look like stack allocations in C are proved nonescaping.
Consequently, while Fil-C does have overheads, this isn’t the one I worry about.
cryptonector 1 hours ago [-]
I see! Thanks for that answer. I'm sure I'll have lots of questions, like these:
You say you don't have to instrument malloc(), but somehow you must learn of the allocation size. How?
Are aliasing bugs detected?
I assume that Fil-C is a whole-program-only option. That is, that you can't mix libraries not compiled with Fil-C and ones compiled with Fil-C. Is that right?
So one might want a whole distro built with Fil-C.
How much are you living with Fil-C? How painful is it, performance-wise?
BTW, I think your approach is remarkable and remarkably interesting. Of course, to some degree this just highlights how bad C (and C++) is (are) at being memory-safe.
pjmlp 5 hours ago [-]
Are you sure they were being used at all?
GCC replaces memcpy/memmove/memset with its own intrisics, if compiling in high optimization levels.
pizlonator 4 hours ago [-]
Yes they were being used.
LukeShu 22 hours ago [-]
When I was working with Envoy Proxy, it was known that perf was worse with musl than with glibc. We went through silly hoops to have a glibc Envoy running in an Alpine (musl) container.
abnercoimbre 1 days ago [-]
Interesting! Will you stick around with the musl build? And if so, why?
pizlonator 23 hours ago [-]
Not sure but in likely to because right now I to use the same libc in userland (the Fil-C compiled part) and yololand (the part compiled by normal C that is below the runtime) and the userland libc is musl.
Having them be the same means that if there is any libc function that is best implemented by having userland call a Fil-C runtime wrapper for the yololand implementation (say because what it’s doing requires platform specific assembly) then I can be sure that the yololand libc really implements that function the same way with all the same corner cases.
But there aren’t many cases of that and they’re hacks that I might someday remove. So I probably won’t have this “libc sandwich” forever
ObscureScience 1 days ago [-]
That table is unfortunately quite old. I can't personally say what have changed, but it is hard to put much confidence in the relevance of the information.
lifthrasiir 23 hours ago [-]
Yeah, also it doesn't compare actual implementations, just plain checkboxes. I'm aware of two specific substantial performance regressions for musl: exact floating point printing (it uses Dragon4 but implemented it way slower than it could have been) and memory allocator (for a long time it didn't any sort of arena like pretty much every modern allocator---now it does with mallocng though).
weiwenhao 10 hours ago [-]
The static compilation of musl libc is a huge help for alpine linux and many system programming languages. My programming language https://github.com/nature-lang/nature is also built on musl libc.
josephg 13 hours ago [-]
It’s amazing how much code gets pulled in for printf. Using musl, printf apparently adds 13kb of code to your binary. Given format strings are almost always static, it’s so weird to me that they still get parsed at runtime in all cases. Modern compilers even parse printf format strings anyway to check your types match.
This sort of thing makes me really appreciate zig’s comptime. Even rust uses a macro for println!().
messe 12 hours ago [-]
In larger programs, that compile time parsing can lead to even more code, as the function is essentially instantiated and compiled separately for each and every invocation. The type erasure provided by printf, can be a blessing in _some circumstances_.
That being said, in those larger programs, it's still likely going to be a negligible part of the binary size, and the additional code paths are unlikely to affect performance unless you're doing string formatting in multiple hot-paths which is generally a poor choice anyway.
jcelerier 12 hours ago [-]
If you use any level of compiler optimisation both clang and GCC will convert calls to printf into calls to puts (which is much simpler) if they detect there's no formatting done
thrtythreeforty 1 days ago [-]
It really ought to lead with the license of each library. I was considering dietlibc until I got to the bottom - GPLv2. I am a GPL apologist and even I can appreciate that this is a nonstarter; even GNU's libc is only LGPL!
LeFantome 1 days ago [-]
musl seems to have displaced dietLibc. Much more complete yet fairly small and light.
yusina 1 days ago [-]
Note that dietlibc is the project of a sole coder in the CCC sphere from Berlin (Fefe). His main objective was to learn how low level infra is implemented and started using it in some of his other projects after realizing that there is a lot of bloat he can skip with just implementing the bare essentials. Musl has a different set of objectives.
projektfu 21 hours ago [-]
I follow diet but it is definitely not ready for general use like musl and probably never will be. There aren't a lot of eyeballs on it.
yusina 15 hours ago [-]
That's what I'm saying. It's not Fefe's objective to make it fit for everybody...
13 hours ago [-]
moomin 1 days ago [-]
No cosmopolitan, pity.
jay-barronville 1 days ago [-]
Please note that the linked comparison table has been unmaintained for a while. This is even explicitly stated on the legacy musl libc website[0][0] (i.e., “The (mostly unmaintained) libc comparison is still available on etalabs.net.”).
This comparison was last updated around 2016-2017. Since then, glibc has improved its size efficiency (particularly with link-time optimization), musl has enhanced its POSIX compliance, and several performance optimizations have landed in both projects.
snickerer 1 days ago [-]
Fun libc comparison by the author of musl.
My getaway is: glibc is bloated but fast. Quite unexpected combination. Am I right?
kstrauser 1 days ago [-]
It’s not shocking. More complex implementations using more sophisticated algorithms can be faster. That’s not always true, but it often is. For example, look at some of the string search algorithms used by things like ripgrep. They’re way more complex than just looping across the input and matching character by character, and they pay off.
Something like glibc has had decades to swap in complex, fast code for simple-looking functions.
weinzierl 1 days ago [-]
In case of glibc I think what you said is orthogonal to its bloat. Yes, it has complex implementations but since they are for a good reason I'd hardly call them bloat.
Independently from that glibc implements a lot of stuff that could be considered bloat:
- Extensive internationalization support
- Extensive backward compatibility
- Support for numerous architectures and platforms
- Comprehensive implementations of optional standards
kstrauser 1 days ago [-]
Ok, fair points, although internationalization seems like a reasonable thing to include at first glance.
Is there a fork of glibc that strips ancient or bizarre platforms?
SAI_Peregrinus 1 days ago [-]
It's called glibc. Essentially all that "bloat" is conditionally compiled, if your target isn't an ancient or bizarre platform it won't get included in the runtime.
kstrauser 1 days ago [-]
That’s mostly true, but not quite. For instance, suppose you aim to support all of 32/64-bit and little/big-endian. You’ll likely end up factoring straightforward math operations out into standalone functions. Granted, those will probably get inlined, but it may mean your structure is more abstracted than it would be otherwise. Just supporting the options has implications.
That’s not the strongest example. I just meant it to be illustrative of the idea.
jcranmer 23 hours ago [-]
The way glibc's source works (for something like math functions) is that essentially every function is implemented in its own file, and various config knobs can provide extra directories to compile and provide function definitions. This can make actually finding the implementation that's going to be used difficult, since a naive search for the function name can turn up like 20 different function definitions, and working out which one is actually in play can be difficult (especially since it's more than just the architecture name).
Math functions aren't going to be strongly impacted by diverse hardware support. In practice, you largely care about 32-bit and 64-bit IEEE 754 types, which means your macros to decompose floating-point types to their constituent sign/exponent/significand fields are already going to be pretty portable even across different endianness (just bitcast to a uint32_t/uint64_t, and all of the shift logic will remain the same). And there's not much reason to vary the implementation except to take advantage of hardware instructions that implement the math functions directly... which are generally better handled by the compiler anyways.
saagarjha 17 hours ago [-]
People don't typically implement math functions by pulling bits out of a reinterpreted floating point number. If you rely on the compiler, you get whatever it decides for you, which might be something dumb like float80.
int_19h 14 hours ago [-]
"Internationalization" is a very broad item that can include e.g. support for non-UTF-8 locales, which is something few Linux distros need today.
dima55 1 days ago [-]
What problem are you trying to solve? glibc works just fine for most use cases. If you have some niche requirements, you have alternative libraries you can use (listed in the article). Forking glibc in the way you describe is literally pointless
kstrauser 1 days ago [-]
Nothing really. I was just curious and this isn’t something I know much about, but would like to learn more of.
A lot of the “slowness” of MUSL is the default allocator. It can be swapped out.
For example, Chimera Linux uses MUSL with mimalloc and it is quite snappy.
jeffbee 22 hours ago [-]
That's a great combo. I like LLVM libc in overlay mode with musl beneath and mimalloc. Performance is excellent.
userbinator 12 hours ago [-]
Microbenchmarks tend to favour extreme unrolling and other "speed at any cost" tricks that often show up as negatives in macrobenchmarks.
timeinput 1 days ago [-]
My take away is that it's not a meaningful chart? Just in the first row musl looks bloated at 426k compared to dietlibc at 120k. Why were those colors chosen? It's arbitrary and up to the author of the chart.
The author of musl made a chart, that focused on the things they cared about and benchmarked them, and found that for the things they prioritized they were better than other standard library implementations (at least from counting green rows)? neat.
I mean I'm glad they made the library, that it's useful, and that it's meeting the goals they set out to solve, but what would the same chart created by the other library authors look like?
cyberax 1 days ago [-]
Not quite correct. glibc is slow if you need to be able to fork quickly.
However, it does have super-optimized string/memory functions. There are highly optimized assembly language implementations of them that use SIMD for dozens of different CPUs.
casey2 17 hours ago [-]
Where is the "# of regressions caused" box?
edam 1 days ago [-]
Pretty obviously made by the musl authors.
deaddodo 1 days ago [-]
> "I have tried to be fair and objective, but as I am the author of musl"
Yeah, pretty obvious when they state as much in the first paragraph.
What's yoyoland? All I can find is an amusement park in Bangkok, and some 1990s-era communication software for Classic Mac OS: https://www.macintoshrepository.org/39495-yoyo-2-1
- Userland: the place where you C code lives. Like the normal userland you're familiar with, but everything is compiled with Fil-C, so it's memory safe.
- Yololand: the place where Fil-C's runtime lives. Fil-C's runtime is about 100,000 lines of C code (almost entirely written by me), which currently has libc as a dependency (because the runtime makes syscalls using the normal C functions for syscalls rather than using assembly directly; also the runtime relies on a handful of libc utility functions that aren't syscalls, like memcpy).
So Fil-C has two libc's. The yololand libc (compiled with a normal C compiler, only there to support the runtime) and the userland libc (compiled with the Fil-C compiler like everything else in Fil-C userland, and this is what your C code calls into).
On Linux, if all you need is syscalls, you can just write your own syscall wrapper-like Go does.
Doesn’t work on some other operating systems (e.g. Solaris/Illumos, OpenBSD, macOS, Windows) where the system call interface is private to the system shared libraries
Unless you do special things, the compiler turns __builtin_memcpy into a call to memcpy. :-)
There is __builtin_memcpy_inline, but then you're at the compiler's whims. I don't think I want that.
A faithful implementation of what you're proposing would have the Fil-C runtime provide a memcpy function so that whenever the compiler wants to call memcpy, it will call that function.
> On Linux, if all you need is syscalls, you can just write your own syscall wrapper-like Go does.
I could do that. I just don't, right now.
You're totally right that I could remove the yolo libc. This is one of like 1,000 reasons why Fil-C is slower than it needs to be right now. It's a young project so it has lots of this kind of "expedient engineering".
I needed a fun term to refer to the C that isn’t Fil-C. I call it Yolo-C.
Hence yololand - the part of the Fil-C process that contains a bit of Yolo-C code for the Fil-C runtime.
> It's even possible to allocate memory using malloc from within a signal handler (which is necessary because Fil-C heap-allocates stack allocations).
Hmm, really? All stack allocations are heap-allocated? Doesn't that make Fil-C super slow? Is there no way to do stack allocation? Or did I misread what you meant by 'stack allocations'?
And that GC allocation only happens if the compiler can’t prove that it’s nonescaping. The overwhelming majority of what look like stack allocations in C are proved nonescaping.
Consequently, while Fil-C does have overheads, this isn’t the one I worry about.
You say you don't have to instrument malloc(), but somehow you must learn of the allocation size. How?
Are aliasing bugs detected?
I assume that Fil-C is a whole-program-only option. That is, that you can't mix libraries not compiled with Fil-C and ones compiled with Fil-C. Is that right?
So one might want a whole distro built with Fil-C.
How much are you living with Fil-C? How painful is it, performance-wise?
BTW, I think your approach is remarkable and remarkably interesting. Of course, to some degree this just highlights how bad C (and C++) is (are) at being memory-safe.
GCC replaces memcpy/memmove/memset with its own intrisics, if compiling in high optimization levels.
Having them be the same means that if there is any libc function that is best implemented by having userland call a Fil-C runtime wrapper for the yololand implementation (say because what it’s doing requires platform specific assembly) then I can be sure that the yololand libc really implements that function the same way with all the same corner cases.
But there aren’t many cases of that and they’re hacks that I might someday remove. So I probably won’t have this “libc sandwich” forever
This sort of thing makes me really appreciate zig’s comptime. Even rust uses a macro for println!().
That being said, in those larger programs, it's still likely going to be a negligible part of the binary size, and the additional code paths are unlikely to affect performance unless you're doing string formatting in multiple hot-paths which is generally a poor choice anyway.
[0]: https://www.musl-libc.org
My getaway is: glibc is bloated but fast. Quite unexpected combination. Am I right?
Something like glibc has had decades to swap in complex, fast code for simple-looking functions.
Independently from that glibc implements a lot of stuff that could be considered bloat:
- Extensive internationalization support
- Extensive backward compatibility
- Support for numerous architectures and platforms
- Comprehensive implementations of optional standards
Is there a fork of glibc that strips ancient or bizarre platforms?
That’s not the strongest example. I just meant it to be illustrative of the idea.
Math functions aren't going to be strongly impacted by diverse hardware support. In practice, you largely care about 32-bit and 64-bit IEEE 754 types, which means your macros to decompose floating-point types to their constituent sign/exponent/significand fields are already going to be pretty portable even across different endianness (just bitcast to a uint32_t/uint64_t, and all of the shift logic will remain the same). And there's not much reason to vary the implementation except to take advantage of hardware instructions that implement the math functions directly... which are generally better handled by the compiler anyways.
https://github.com/lattera/glibc/blob/master/string/strlen.c
For example, Chimera Linux uses MUSL with mimalloc and it is quite snappy.
The author of musl made a chart, that focused on the things they cared about and benchmarked them, and found that for the things they prioritized they were better than other standard library implementations (at least from counting green rows)? neat.
I mean I'm glad they made the library, that it's useful, and that it's meeting the goals they set out to solve, but what would the same chart created by the other library authors look like?
However, it does have super-optimized string/memory functions. There are highly optimized assembly language implementations of them that use SIMD for dozens of different CPUs.
Yeah, pretty obvious when they state as much in the first paragraph.