Generative machine learning (ML) for materials discovery requires a way of representing a chemical system (an encoding) which is compatible with modern ML algorithms. To facilitate training, the encoding should satisfy the conditions of invertibility (that the encoding can be interconverted between human-interpretable and machine-interpretable formats without loss of information) and invariance (that each unique structure corresponds to a unique encoding irrespective of coordinate transformation, preventing data leakage and accelerating training). While invertible, invariant representations for molecules such as SMILES have been developed and deployed (e.g. in drug design and discovery), this remains an open problem for periodic structures, where periodic boundary conditions add unique challenges. Current state-of-the-art methods for periodic structures explicitly encode the lattice vectors and/or periodic boundary, and are therefore not invariant to the choice/orientation of lattice vectors. Here, we present an alternative approach where geometry is encoded as internal coordinates (bonds, angles, torsions) while lattice vectors and periodic boundaries are not encoded. This approach ensures full roto-translational invariance, while lattice vectors and periodic boundary conditions remain recoverable as an emergent property of the underlying system. Generation of novel materials under this encoding entails sampling the composition (number and type of atoms), connectivity (number and type of bonds) and geometry (encoded as internal coordinates). We adopt a modular approach, with independent models trained for each generative task. The design and implementation of the graph encoding and decoding tool will be discussed, as well as the development of diffusion-based connectivity and geometry models. Limitations of and avenues for future developments of both the encoding and generative models will be discussed.