Proposal: Private Fields

Introduction

Currently, function and variable declarations are private by default and are made externally visible with the pub modifier. However, data fields are always public and there is no way to restrict their visibility. Apparently, field access control was never really working in Zig (#569) and was officially removed in #2059. Which is somewhat surprising, since no other modern language I can think of (Java, C++, C#, D, Rust, Go, ...) fails to provide this feature.

The lack of private data makes it impossible to do proper encapsulation / implementation hiding, which is widely considered to be a basic software engineering technique. In particular:

• Types can hide some of their methods and constants, but always expose their internal structure. This leaves them open to accidental (or bad-practical) corruption. The compiler will not complain if the .capacity of an ArrayList is overwritten.
• Documentation value is lost. Whether or not a field is intended for direct access can only be communicated through code comments.
• Auditing code becomes more difficult, because there are more opportunities to violate constraints and invariants.
• Implementing opaque handles is nearly impossible without additional builtins: Introduce @OpaqueHandle() builtin #9859, #1595 (comment).

Possible objections

No official reason was given in #2059, but two possible objections to private fields come to mind:

1. Visibility control is more useful for decls than for fields, because invisible decls become completely inaccessible, while fields can always be messed with through pointers.
2. This will lead to over-encapsulation and getter-setter boilerplate (#2974 (comment)). Sometimes it's just better to reach in and do things directly.

Concerning 1, I'd say that protection does not have to be perfect to be useful. Preventing accidental and semi-deliberate messing is valuable by itself, not to mention the documentation value.

2 may be a real concern, or maybe not. The same argument can be made about private decls, but a proposal to override the visibility of methods at the call site (#8779) was recently met with a resounding rejection -- even though there are certainly cases where it is both reasonable and safe to call private methods. In addition, Zig is increasingly making legal but potentially buggy patterns into hard errors, somtimes controversially (e.g. unused variables). The lack of basic implementation hiding is not consistent with this safety-first attitude, IMHO.

Syntax

The simplest option is to adopt the same default as with decls: file-level private by default and externally visible with pub. If this clashes with data-oriented style, the opposite default can be chosen, but that wold require the introduction of a private keyword. Yet another possibility is to make field visibility struct-level, e.g. with opaque struct {...}.

I don't really have a strong opinion here.

Open questions

• Is there a case for field-level access control in unions?

---

Update 1: As a side benefit, it may be possible to remove the opaque {} type from the language, since it is rarely used and can be simulated with struct { private ptr: usize }.

Update 2: I wouldn't mind adding an escape hatch like @privateField(object, "name") for cases where you need to use a particular library, but find the API too locked-down. Some discussion of this is here.

[vijfhoek]

I feel the argument you make in the objections ("In addition, Zig is increasingly making legal but potentially buggy patterns into hard errors, somtimes controversially (e.g. unused variables).") is a very good one. This is something that I feel Zig has evolved in since #2059.

Maybe worth moving that argument up to the introduction, to pay it more attention?

A possible objection is that this could also be achieved with a strict naming convention, e.g. starting variables with an underscore will mark them as if they're private, and perhaps be enforced with future linter rules or the like. Just a thought.

[batiati]

To support private fields, from zig zen:

• Communicate intent precisely.

[rohlem]

As for documentation value, in status quo I would suggest:

pub const ByteList = struct {
 // public decls and members here
 data: []u8,

 private: ByteListPrivate,
};
/// non-public, private decls and members here
const ByteListPrivate = struct {
  allocator: std.mem.Allocator,
  capacity: usize,
};

Accessing fields in .private should not occur accidentally, and be relatively easy to spot in a code review.
If you use reflection to iterate over fields, maybe have a private-informed branch of std.meta to exclude/ warn about specific field names.

I'll probably have to reread the @OpaqueHandle discussion, but it should be possible to replace the field with an equally-sized-and-aligned byte array align(@alignOf(Private)) private: [@sizeOf(Private)]u8,.
Then @ptrCast/@bitCast back-and-forth with non-public helper functions.

The missing piece then is "making the compiler complain" when overwriting the .private field.
The one scenario that comes to my mind would be reassigning the entire containing struct (ByteList above).
Maybe declaring the .private field to be pinned, as in #7769, would already be enough of a solution - granted though, the actual source of the error would be the fact you can't copy from .private, instead of inability to copy to .private.

This might be really useful, it doesn'make sense to me why access is a concern for decls, but not fields. It should be either both or neither.

Just an opinion:
Maybe its private fields that should be marked as such, not public?

As for setter/getter boilerplate, half of a problem can possibly be solved with something like 'readonly' keyword, which would effectively act as '*const' outside of a file scope.

@rohlem
OpaqueHandle would indeed provide equivalent functionality: You factor out your private state into a separate struct, wrap it in an @OpaqueHandle, and then use @toOpaqueHandle() and @fromOpaqueHandle() to read and write. My question is, why use such a roundabout way to hide state, when there is the well-established technique of visibility modifiers?

Also, and somewhat ironically, the OpaqueHandle trick only works because it leverages the existing ability to declare a private struct. Otherwise there would be very little point in the whole concept of OpaqueHandle, and you might just as well go with a naming convention like _private, as @vijfhoek suggested.

[Guigui220D]

I have been using naming conventions for my private fields, like _capacity: usize in order to discourage usage, but I feel like having pub for fields would be better.

[InKryption]

Personally, I quite like that data in zig is so transparent; it makes no attempt to model anything other than what the computer is really doing.

That said, there is obviously a use case for data hiding, but I don't think it should be as easy as just marking a field as private; and as well, I think this shouldn't be as powerful as the privatization of declarations, and moreso act as stronger documentation than comments. Essentially, I think we should continue to encourage transparent data, by making private fields harder to declare in some way.

In that regard, I actually think something similar to what was modelled in #9859 would actually be better for this purpose - using a separate "opaque bag of bits" as a section for private data. This would not only slightly discourage private data, but also document even better the separation of private and public fields.

I think there is perhaps also an argument to be made that, if private fields, whether in the proposed style or in the #9859 style, maybe some form of controlled access into private fields should be enabled anyway, like a built-in (e.g. @privateField(expr, "private field name")). Because unlike what was argued against @call being able to access private functions, accessing what is just data wouldn't be as possibly destructive to any "library state" - and as well, as has been said in this proposal, even with private fields, we still have the ability to mess with the data, so maybe granting an assuredly safe mechanism to do so in lieu of "smartness" would be a good move (and as a bonus, it would be a clear mark of possible code-smell, versus aforementioned smartness).

Just some food for thought. I'll stop typing now.

Personally, I quite like that data in zig is so transparent.

So do I, and in private projects I don't usually bother with access restrictions, even in other languages. But is this sufficient reason to make data hiding inconvenient or impossible in places where it is needed (e.g. public-facing libraries or high-assurance projects)?

In that regard, I actually think something similar to what was modelled in #9859 would actually be better for this purpose - using a separate "opaque bag of bits" as a section for private data. This would not only slightly discourage private data, but also document even better the separation of private and public fields.

I'll have to disagree here. To me, pub or private is as clear documentation as it gets for the intended access level. Putting everything into a single private section may also interfere with data layout, although that is a secondary concern.

As for adding a @privateField override, I'm all for it. Like you say, this indicates code smell clearly enough, and should allay fears along the lines of "other programmers will abuse private fields and will only get in my way". Though I don't see why private decls shouldn't be accessible in the same fashion. While reading private data is not as bad as calling private methods, modifying it is probably worse. Also, both would be needed for "extending" library containers, for example.

[InKryption]

@zzyxyzz As it pertains to private declarations, I think the most convincing rationale I've heard for it is that it forces change at an API level, as opposed to forcing it at the library-usage level; that is to say, if a library has marked a certain function as private, but the community at large has decided that it would be best marked public, this will either force the library implementers to push the change, or force a fork of the library that implements this change (with the former being the preferred outcome). This as opposed to forcing people wrapping said library to use functions that don't have the same guarantees as those explicitly marked public.

I can't quite recall where I read that here, or whether or not that's a good enough reason. But for the purposes of arguing against applying this logic it to private fields, at the very least, I would point to the fact that although a field may be marked private, it cannot be guaranteed to be inaccessible, where its memory is accessible in the public-facing API (as explained), and thus it makes sense to treat field access modifiers more like built-in documentation tools, as opposed to strict API boundary controls.

(Please tell me if any of this makes sense, my synthesizing skills are not pristine).

@InKryption

that is to say, if a library has marked a certain function as private, but the community at large has decided that it would be best marked public, this will either force the library implementers to push the change, or force a fork of the library that implements this change (with the former being the preferred outcome).

I've heard this argument too, but I find it unrealistically idealistic. Even assuming that your proposed change will be accepted into the public API (which will not happen most of the time, either because your use-case is too specific or the authors plain disagree with your way of doing things), the usual timeline for the change is measured in months or years. And outright forks rarely happen at all, unless the project is abandoned by the previous author. So, while this position does represent best practice for upstreaming changes in open-source projects, it has little to no relevance for everyday software development.

[InKryption]

I am glad that we generally agree, then. I'm just bringing the argument to the forefront.

[sina-devel]

If the private keyword is added, it's not syntactically compatible (functions, etc are private by default and exported by the pub keyword)

[InKryption]

@sina-devel I would argue that it makes sense for functions to private by default, as it incentivizes explicitly building a robust public-facing API, as opposed to simply allowing everything to spill into global scope. Whereas, here, I think it is more useful to overall make transparent data the default, and incentivize having to explicitly decide if data should really be marked as an "internal" component (which would likely lead to more boilerplate related to reading and writing to the data, which should certainly not be the default, in my opinion, coming from a C++ background).

[sina-devel]

adding private or internal:

• boilerplate code for geter (e.g. geter for capacity of ArrayList)
• stdlib changes a lot
• ...

right now:

• all fields are public for get and set

by default private and exported by pub:

• boilerplate code for geter (e.g. geter for capacity of ArrayList)
• stdlib changes a lot
• ...

adding keyword to readonly at outside and mutable at inside:

• complexity increases
• ...