Ubuntu – Create csv from inconsistent text file

A Perl approach:

$ perl -lne 'next unless /^\s*\*\s*(\S+)/; $n=$1; @k=(/\{.+?:\s*\[(.+?)\]/g); print "$n @k"' file 
test_repo_one/xenial xenial-main_20190311 xenial-multiverse_20190311 xenial-restricted_20190311 xenial-universe_20190311
test_repo_one/xenial-security xenial-security-main_20190311 xenial-security-multiverse_20190311 xenial-security-restricted_20190311 xenial-security-universe_20190311
test_repo_two/trusty trusty-main_20190312 trusty-multiverse_20190312 trusty-restricted_20190312 trusty-universe_20190312

Explanation

• perl -lne: read the input file line by line (-n), remove trailing newlines (-l) and run the script given by -e on each line. The -l also adds a newline to each print call.
• next unless /^\s*\*\s*(\S+)/; : find the name of the repo, so the first stretch of non-whitespace characters (\S+) on a line that starts with 0 or more whitespace characters (^\s*), then a * (\*), and 0 or more whitespace characters again. The longest stretch of non-whitespace after that is what we want. If this line doesn't match this regex, the next will move us onto the next line.
• $n=$1 : save what was captured by the match above (the (\S+) in parentheses, $1) as $n.
• @k=(/\{.+?:\s*\[(.+?)\]/g): find all cases where we have a {, any other characters and then a :, followed by whitespace and a [ and capture anything between the [ and the ]. Save all matching strings in the array @k.
• print "$n @k" : finally, print the name of the repo, the $n, and the array @k from above.

If you prefer to have the square brackets included, you can use:

$ perl -lne 'next unless /^\s*\*\s*(\S+)/; $n=$1; @k=(/\{.+?:\s*(\[.+?\])/g); print "$n @k"' file 
test_repo_one/xenial [xenial-main_20190311] [xenial-multiverse_20190311] [xenial-restricted_20190311] [xenial-universe_20190311]
test_repo_one/xenial-security [xenial-security-main_20190311] [xenial-security-multiverse_20190311] [xenial-security-restricted_20190311] [xenial-security-universe_20190311]
test_repo_two/trusty [trusty-main_20190312] [trusty-multiverse_20190312] [trusty-restricted_20190312] [trusty-universe_20190312]

You can try

paste -d, - - - < file.txt > file.csv

The paste command generally merges lines of files side-by-side (with an optional delimiter supplied by -d). In this case we are telling it to read the standard input stream 3 times - - - instead.