iter.Seq in practice
Jarrod RobersonFirstN
Returns the first N items from a sequence.
// FirstN takes an iter.Seq[int] and returns a new iter.Seq[int] that
// yields only the first 'limit' items without creating intermediate slices.
func FirstN[T any](original iter.Seq[T], limit int) iter.Seq[T] {
return iter.Seq[T](func(yield func(T) bool) {
count := 0
for item := range original {
if count < limit {
if !yield(item) {
return
}
count++
} else {
return
}
}
})
}
SkipFirstN
Skips the first N items and returns the rest of the sequence.
This could have been done with a regular push iterator, but it would take an intermediate count to keep up with how many items had been skipped and not be quiet as clear what was happening.
func SkipFirstN[T any](seq iter.Seq[T], skip int) iter.Seq[T] {
return iter.Seq[T](func(yield func(T) bool) {
next, stop := iter.Pull[T](seq)
defer stop()
for i := 0; i <= skip; i++ {
_, ok := next()
if !ok {
break
}
}
for {
v, ok := next()
if !ok {
break
}
if !yield(v) {
return
}
}
})
}
SkipAndLimit (aka: SubSeq)
Example inline composition of FirstN and SkipFirstN this is just an example.
This is the equivalent of doing [first:limit] on a slice but on an iter.Seq as it is processed as a zero cost abstraction.
// I wrapped this in a function as straw man example just to get the highlighting to work
func SkipAndLimit[V any](it iter.Seq[V],, skip int, limit int) iter.Seq[V] {
return FirstN[V](SkipFirstN[V](it, skip), limit)
}
// you could just inline the composition like this, but this is not as self documenting
// as calling a function that semantically shows intent in its name.
subSeq := FirstN[string](SkipFirstN[string](it, skip, limit)
Chunk
creates a sequence of fixed size sequences from the original sequence without creating any intermediate slices or arrays.
This is useful when you are feeding an api data that only accepts N number of items at time. With some work this could be made to work with goroutines and channels to process the chunks in parallel as they are generated.
// Chunk returns an iterator over consecutive sub-slices of up to n elements of s.
// All but the last iter.Seq chunk will have size n.
// Chunk panics if n is less than 1.
func Chunk[T any](sq iter.Seq[T], size int) iter.Seq[iter.Seq[T]] {
if size < 0 {
panic(errs.MinSizeExceededError.New("size %d must be >= 0", size))
}
return func(yield func(s iter.Seq[T]) bool) {
next, stop := iter.Pull[T](sq)
defer stop()
endOfSeq := false
for !endOfSeq {
// get the first item for the chunk
v, ok := next()
// there are no more items !ok then exit loop
// this prevents returning an extra empty iter.Seq at end of Seq
if !ok {
break
}
// create the next sequence chunk
iterSeqChunk := func(yield func(T) bool) {
i := 0
for ; i < size; i++ {
if ok {
if !ok {
// end of original sequence
// this sequence may be <= size
endOfSeq = true
break
}
if !yield(v) {
return
}
v, ok = next()
}
}
}
if !yield(iterSeqChunk) {
return
}
}
}
}
Chunk2
Same idea as Chunk but works on iter.Seq2
// Chunk2 returns an iterator over consecutive sub-slices of up to n elements of s.
// All but the last iter.Seq chunk will have size n.
// Chunk2 panics if n is less than 1.
func Chunk2[K any, V any](sq iter.Seq2[K, V], size int) iter.Seq[iter.Seq2[K, V]] {
if size < 0 {
panic(errs.MinSizeExceededError.New("size %d must be >= 0", size))
}
return func(yield func(s iter.Seq2[K, V]) bool) {
next, stop := iter.Pull2[K, V](sq)
defer stop()
endOfSeq := false
for !endOfSeq {
// get the first item for the chunk
k, v, ok := next()
// there are no more items !ok then exit loop
// this prevents returning an extra empty iter.Seq at end of Seq
if !ok {
break
}
// create the next sequence chunk
iterSeqChunk := func(yield func(K, V) bool) {
i := 0
for ; i < size; i++ {
if ok {
if !ok {
// end of original sequence
// this sequence may be <= size
endOfSeq = true
break
}
if !yield(k, v) {
return
}
k, v, ok = next()
}
}
}
if !yield(iterSeqChunk) {
return
}
}
}
}
SeqToSeq2
This function takes a sequence and a key generator function and returns an iter.Seq2 without creating any intermediate slices or arrays.
func SeqToSeq2[K any, V any](is iter.Seq[V], keyFunc func(v V) K) iter.Seq2[K, V] {
return iter.Seq2[K, V](func(yield func(K, V) bool) {
for v := range is {
k := keyFunc(v)
if !yield(k, v) {
return
}
}
})
}
Map
Most of you probably think I should have put this one first. It is the most similar to a Monad with the bind and map behavior, it is named Map for a reason.
But that would completely miss the point of the iter package. It is not about functional programing, it is about a idiomatic way to create your own custom iterable types.
This is the closest to Monad behavior by definition, skipping the bind semantics and just applying the map function to the value. If your value object had a field called ID that should be the key, the keyFunc would be as simple as returning v.ID or as complex as calculating a SHA256 hash of the object to use as the key. Or something that is not even a map, like returning FirstName and LastName from Person structs.
The fact that the K value is typed any and not comparable means it is not just for keys from maps.
func Map[T any, R any](it iter.Seq[T], mapFunc func(t T) R) iter.Seq[R] {
return func(yield func(R) bool) {
for i := range it {
if !yield(mapFunc(i)) {
return
}
}
}
}
Map2
Does the same thing as Map but allows to provide for both a K and V transform function.
A canonical PassThruFunc for the times when I did not want to change the key or value.
Using this instead of writing in inline is better because it shows intent to not do anything with the value. An inline function shows no intent. Much like “This Page Left Intentionally Blank” shows intent to the reader that they are not missing a page by accident. Very important for everyone reading it in the future, yourself included. It could also be used for the value when only the K needs to be transformed.
// PassThruFunc passes thru the value unchanged
// this is just a convience function for times when you do not want to transform the key or value in Map2
// so you do not have to write an inline function and clutter up the code more than it needs to be.
func PassThruFunc[T any](t T) T {
return t
}
func Map2[K any, V any, KR any, VR any](it iter.Seq2[K, V], keyFunc func(k K) KR, valFunc func(v V) VR) iter.Seq2[KR, VR] {
return func(yield func(KR, VR) bool) {
for k, v := range it {
if !yield(keyFunc(k), valFunc(v)) {
return
}
}
}
}
DocumentIteratorToSeq
Firestore Document Iterator is one of the many non-standard iterator implementations that abound in Go libraries and clients
Here it is wrapped in an iter.Seq without any intermediate slices or arrays being generated.
// DocumentIteratorToSeq converts a firestore.Iterator to an iter.Seq.
// value is a pointer to the type V
func DocumentIteratorToSeq(dsi *fs.DocumentIterator) iter.Seq[*fs.DocumentSnapshot] {
return func(yield func(ref *fs.DocumentSnapshot) bool) {
defer dsi.Stop()
for {
doc, err := dsi.Next()
if errors.Is(err, iterator.Done) {
return
}
if err != nil {
log.Error().Err(err).Msg("error iterating through Firestore documents")
return
}
if !yield(doc) {
return
}
}
}
}
DocSnapShotSeqToType
Here we create a semantically named function that represents something that needs to be done repeatedly. Iterator over a collection of DocumentSnapShots and convert them to actual Types.
The seq.Map function I wrote is usually simple enough to inline, but some cases are so common and repeated it makes sense to semantically name them for ease of use. And no naming it not that hard unless you are not a native English speaker or your function is so poorly designed it can not be named semantically specific.
func DocSnapShotSeqToType[R any](it iter.Seq[*fs.DocumentSnapshot]) iter.Seq[*R] {
return seq.Map[*fs.DocumentSnapshot, *R](it, func(dss *fs.DocumentSnapshot) *R {
var t R
err := dss.DataTo(&t)
if err != nil {
log.Error().Err(err).Msgf("error unmarshalling Firestore document with ID %s", dss.Ref.ID)
panic(err)
}
return &t
})
}
DocumentIteratorToSeq2
Same as above but the DocumentSnapShot.Ref.Id is used as the K value as a string in the returned iter.Seq2.
This is short enough to inline, but I need to do this in lots of places in library code over and over again so making a semantically intuitive named function makes my code clearer and more readable for intent.
I make a point of always using string types for document id so this is used everywhere I process collections.
// DocumentIteratorToSeq2 converts a firestore.Iterator to an iter.Seq2.
// doc.Ref.ID is used as the "key" or first value, second value is a pointer to the type V
func DocumentIteratorToSeq2(dsi *fs.DocumentIterator) iter.Seq2[*fs.DocumentRef, *fs.DocumentSnapshot] {
return seq.SeqToSeq2[*fs.DocumentRef, *fs.DocumentSnapshot](DocumentIteratorToSeq(dsi), func(v *fs.DocumentSnapshot) *fs.DocumentRef {
return v.Ref
})
}
Another iter.Seq2 that is useful for Firestore manipulation is [path, DocumentSnapshot] which is easy to inline as the following and probably warrants its own function if it gets used often enough.
pathDocumentSnapshot := seq.SeqToSeq2[string, *fs.DocumentSnapshot](DocumentIteratorToSeq(dsi), func(v *fs.DocumentSnapshot) string {
return v.Ref.Path
})
Imports
Here is a list of the imports used in the code examples above.
I started using ZeroLog long before slog was added to the standard library and I have gotten spoiled by its features. Most of I rarely use, but when I do need them they are important and not in the slog implementation.
I do not want to have to maintain mixed logging APIs. I had to deal with that in the Java world where sometimes 4 or more different logging libraries were used in a code base and adapters and all that were a nightmare. So, it is easier to just stick with ZeroLog rather than convert all the dependencies I use that I have written to slog.
Errorx is also something I have come to rely on heavily, it makes it really easy to have semantically specific type safe errors so when things breaks in a server with dozens of nested calls in the tree it is not a mystery where, what and why something happened.
import (
"iter"
"slices"
"strings"
fs "cloud.google.com/go/firestore"
"google.golang.org/api/iterator"
"github.com/rs/zerolog/log"
"github.com/joomcode/errorx"
errs "github.com/jarrodhroberson/ossgo/errors"
)
Subscribe to my newsletter
Read articles from Jarrod Roberson directly inside your inbox. Subscribe to the newsletter, and don't miss out.
Written by
Jarrod Roberson
Jarrod Roberson
None of what I post is "ai" generated. "AI" does not exist and what is being called "ai" vomits up misinformation as facts mixed in with a sprinkling of actual facts to make it extremely harmful to use. What you read here, I wrote.