Re: More anti-BPF FUD from Cyberax [LWN.net]

Re: More anti-BPF FUD from Cyberax

Posted Apr 11, 2019 19:35 UTC (Thu) by ecree (guest, #95790)
In reply to: Managing sysctl knobs with BPF by Cyberax
Parent article: Managing sysctl knobs with BPF

> BPF is NOT a language. It's a virtual machine
With the result that there are multiple different languages targeting that backend, from the high-level (bpftrace) through the low (C) right down to assembler (ebpf_asm).

Which in my opinion is a much better approach than every piece of kernel policy having its own virtual machine (netfilter comes to mind). If a particular policy API wants to have its own DSL, it can do that while still using eBPF for the mechanism. And because the DSL is purely a userspace thing, no-one is _forced_ to use it — e.g. when writing functional tests for XDP I found it very helpful to be able to hand-craft the assembler rather than pulling in a gigantic blob of LLVM just to do the same thing in C.

> with a particularly nasty instruction encoding
Firstly, there's nothing nasty about eBPF's encoding; it's cleaner and more orthogonal than most hard ISAs. Secondly, why do you care what the instruction encoding is, unless you're actually writing the toolchain?

> it's not technically Turing-complete
A deliberate design decision, ensuring that eBPF expresses procedurally only those policies that could, at the cost of far greater complexity in both interface and implementation, be expressed declaratively.

> it can be extremely difficult to understand
Only to the extent necessitated by the complexity of policies it allows the user to configure. You can write an eBPF _program_ that's difficult to understand, but that doesn't make eBPF _itself_ difficult to understand.

> Meanwhile, AppArmor policies are simple text files which are self-explanatory.
Which is why everyone is happily using AppArmor and never felt the need to invent cgroups and BPF-based security. Oh wait...

to post comments

Re: More anti-BPF FUD from Cyberax

Posted Apr 11, 2019 20:00 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (13 responses)

> Which in my opinion is a much better approach than every piece of kernel policy having its own virtual machine (netfilter comes to mind). If a particular policy API wants to have its own DSL, it can do that while still using eBPF for the mechanism. And because the DSL is purely a userspace thing, no-one is _forced_ to use it — e.g. when writing functional tests for XDP I found it very helpful to be able to hand-craft the assembler rather than pulling in a gigantic blob of LLVM just to do the same thing in C.
You have a production machine. You have a problem (some software doesn't work). There is no source code for BPF. How are you going to debug it? Can you single-step BPFs attached to /proc files? Is there a way to get an execution trace?

I have already had a wonderful opportunity to do exactly this. IT. WAS. NOT. FUN.

BPF violates the cardinal rule of software design - your software must be easy to troubleshoot.

> Which is why everyone is happily using AppArmor and never felt the need to invent cgroups and BPF-based security. Oh wait...
cgroups are not a replacement for AppArmor.

Re: More anti-BPF FUD from Cyberax

Posted Apr 11, 2019 20:51 UTC (Thu) by ecree (guest, #95790) [Link] (9 responses)

> There is no source code for BPF. How are you going to debug it?
With BTF, soon BPF programs will come with complete source-level debuginfo. Also, if you're going around installing BPF programs without having the corresponding source, you're naturally going to have all the same problems as if you drop an object file in /bin and lose the source; how is that BPF's fault?

> Can you single-step BPFs attached to /proc files?
Can you single-step an LSM implementation? Both are just "some executable bytes in the kernel", and debugging facilities have to be built. In the case of LSM, they have to be written into the kernel module, whereas with BPF, the user can insert their own debugging (it is not impossible, for instance, to patch bpf_trace_printk()s into a program that you only have in binary form).

> cgroups are not a replacement for AppArmor.
I wasn't saying they were.

I was merely saying that AppArmor is not a replacement for cgroup-bpf either, because if it was, cgroup-bpf wouldn't have been invented.

Re: More anti-BPF FUD from Cyberax

Posted Apr 11, 2019 21:34 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (8 responses)

> With BTF, soon BPF programs will come with complete source-level debuginfo. Also, if you're going around installing BPF programs without having the corresponding source, you're naturally going to have all the same problems as if you drop an object file in /bin and lose the source; how is that BPF's fault?
I'm not installing anything. I'm debugging my software not working on a particular version of the fleet that is controlled by another team. Or I'm debugging a server after its administrator had died from a heroin overdose (real example).

> drop an object file in /bin and lose the source; how is that BPF's fault?
I can inspect binaries with GDB, I can see their activity with strace(), I can look for source code in repositories, etc.

None of this is possible with BPF.

> Can you single-step an LSM implementation?
Yes. Not quite single step, but explain failures: https://wiki.debian.org/AppArmor/Debug

> I was merely saying that AppArmor is not a replacement for cgroup-bpf either, because if it was, cgroup-bpf wouldn't have been invented.
BPF should have never been implemented at all.

Re: More anti-BPF FUD from Cyberax

Posted Apr 11, 2019 22:22 UTC (Thu) by ecree (guest, #95790) [Link] (7 responses)

> I'm debugging my software not working on a particular version of the fleet that is controlled by another team.

That sounds like "your organisation is dysfunctional", not "BPF is bad".

> I can inspect binaries with GDB, I can see their activity with strace(), I can look for source code in repositories, etc.
>
> None of this is possible with BPF.

You can inspect the disassembly of the BPF program (in future including, as I said, BTF debuginfos); you can debug them with a kernel debugger if you really need to (directly analogous to an LSM; both run in kernel space); you can look for the BPF program's source code in repositories too (e.g. it might be https://github.com/cilium/cilium that's running).

> Yes. Not quite single step, but explain failures

So, no, then. All AppArmor can do is tell you what syscalls etc. it denied; a BPF security program can in principle do the same thing (there may not be built-in mechanisms to do that right now, but there's nothing intrinsic about the BPF model that prevents them, and in the meantime there's always bpf_trace_printk()).

> BPF should have never been implemented at all.

I'd be interested to hear what you think should replace XDP, then, as doubtless would Cloudflare and Facebook.

Brendan Gregg might also have a thing or two to say about tracing. (You're the one who was saying debuggability was so important!)

Re: More anti-BPF FUD from Cyberax

Posted Apr 11, 2019 22:27 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (6 responses)

> That sounds like "your organisation is dysfunctional", not "BPF is bad".
As most large organizations...

> You can inspect the disassembly of the BPF program (in future including, as I said, BTF debuginfos); you can debug them with a kernel debugger if you really need to (directly analogous to an LSM; both run in kernel space);
How can I do this? Is there a tutorial for it?

> So, no, then. All AppArmor can do is tell you what syscalls etc. it denied; a BPF security program can in principle do the same thing
I'm not interested in principles. Is there such an infrastructure right now? If not, then it needs to be a pre-requisite for the kernel merge.

> I'd be interested to hear what you think should replace XDP, then, as doubtless would Cloudflare and Facebook.
I wouldn't mind if it just stays there. I don't care about it being used outside of the network stack.

Re: More anti-BPF FUD from Cyberax

Posted Apr 11, 2019 22:51 UTC (Thu) by ecree (guest, #95790) [Link] (5 responses)

> As most large organizations...

Large organisations are poor design, just as large programs are.

> How can I do this? Is there a tutorial for it?

If the documentation is inadequate or unclear, pester the BPF core developers to improve it. On that, I'll side with you — I had to badger them repeatedly to write a spec for BTF.

> I'm not interested in principles.

Ok, so you're just interested in spreading FUD, then. Your problem isn't with BPF as a design or a concept.

> If not, then it needs to be a pre-requisite for the kernel merge.

Now you're being silly. It might be a pre-requisite for it being a good idea to use it in your organisation; but an opt-in feature doesn't have to be perfect to be merged, it just has to have a stable ABI. Which BPF does; adding verdict tracing wouldn't require changes to the existing ABI.

Re: More anti-BPF FUD from Cyberax

Posted Apr 11, 2019 23:03 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (4 responses)

> Large organisations are poor design, just as large programs are.
Sure. But you can't avoid them and they do a lot of things well.

> If the documentation is inadequate or unclear, pester the BPF core developers to improve it. On that, I'll side with you — I had to badger them repeatedly to write a spec for BTF.
I'm doing this.

> Ok, so you're just interested in spreading FUD, then. Your problem isn't with BPF as a design or a concept.
In the end everything is Turing machine approximation. So yes, my problems are with this particular implementation. It's not acceptable to cause regressions in practical usability.

> Now you're being silly. It might be a pre-requisite for it being a good idea to use it in your organisation; but an opt-in feature doesn't have to be perfect to be merged
It has to be reasonable. The mainline kernel is not a dumping ground for random pet features.

Re: More anti-BPF FUD from Cyberax

Posted Apr 12, 2019 1:13 UTC (Fri) by rahulsundaram (subscriber, #21946) [Link]

> It has to be reasonable. The mainline kernel is not a dumping ground for random pet features.

In many ways, that's precisely how it is.

There is no overarching kernel design. It is mostly a collection of random pet features. If the pet features happened to be usable and useful in ways you don't like, then tough luck.

Re: More anti-BPF FUD from Cyberax

Posted Apr 15, 2019 21:00 UTC (Mon) by togga (subscriber, #53103) [Link] (2 responses)

> It's not acceptable to cause regressions in practical usability.

BPF adds features here, it doesn't cause regressions to anything. Besides, there is an BPF flag when you build your kernel you can say no to.

Re: More anti-BPF FUD from Cyberax

Posted Apr 15, 2019 21:17 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

> BPF adds features here, it doesn't cause regressions to anything. Besides, there is an BPF flag when you build your kernel you can say no to.
"It's an option" is not an excuse for a broken feature.

Re: More anti-BPF FUD from Cyberax

Posted Apr 16, 2019 18:58 UTC (Tue) by togga (subscriber, #53103) [Link]

> "It's an option" is not an excuse for a broken feature.
I just don't see how this feature is broken, it opens up lots of features/possibilities. Hopefully there's good filtering and sense when reviewing/accepting new features and requires enough thought by users so it doesn't derail (look at Python for an example of that).

Re: More anti-BPF FUD from Cyberax

Posted Apr 15, 2019 20:50 UTC (Mon) by togga (subscriber, #53103) [Link] (2 responses)

> There is no source code for BPF. How are you going to debug it?

How do you debug AppArmor? Is that fun?

BPF might be hard to debug today, but the alternative is extending features to a number of DSL:s and specialized modules for a subset of possibilities.

BPF should at least be straightforward to simulate (or run as is) in user-space (think regression-tests) and once you crack this debug-nut, you'd be set for the BPF invasion.

Re: More anti-BPF FUD from Cyberax

Posted Apr 15, 2019 20:58 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

> How do you debug AppArmor? Is that fun?
You look at the logs and see which rule caused the request to fail.

> BPF might be hard to debug today, but the alternative is extending features to a number of DSL:s and specialized modules for a subset of possibilities.
BPF will always be hard to debug, it's almost a general-purpose VM and you'll have to debug it at basically the assembly level.

> BPF should at least be straightforward to simulate (or run as is) in user-space (think regression-tests) and once you crack this debug-nut, you'd be set for the BPF invasion.
Right now there are no tools for simulation. There are no tools to create BPF filters as well (nothing like "learning mode" in AppArmor and SELinux).

Re: More anti-BPF FUD from Cyberax

Posted Apr 16, 2019 18:47 UTC (Tue) by togga (subscriber, #53103) [Link]

> You look at the logs and see which rule caused the request to fail.
Yes. This is like looking at BPF printk's and perf events to user-space right? At assembly-level AppArmor is probably worse?

> BPF will always be hard to debug, it's almost a general-purpose VM and you'll have to debug it at basically the assembly level.
Correct. At least the assembly level is much easier here than for the AppArmor case since it's higher level. Complexity with the JIT could be an issue however. Another issue with BPF could be the usual effects of "no taste/craftsmanship" when given flexibility (over-engineering at one end and the tunnel-visioned mess on the other).

> Right now there are no tools for simulation. There are no tools to create BPF filters as well (nothing like "learning mode" in AppArmor and SELinux).
Lots of opportunities here :-)