suggestion/bug fix for "readDBFile"

This has nothing to do with OS, browser or network.  It's a feature request for the readDBFile procedure.

My app reads a semicolon delimited text database.

If a record is generated incorrectly (in this case, it had double quote marks in it), the "readDBFile" command dies and tells you:

Syntax Error: quote in unquoted cell" & "Cannot parse 
text argument to "list from csv table: as a CSV-formatted table"

I have two requests: 

1. SHOW THE LINE NUMBER WHERE THE PARSE FAILED. This makes debugging a hell of a lot easier. 

2. SKIP THE BAD LINE AND KEEP READING the database. Announce the bad line numbers at the end of readDBFile. 

If you want, abort the readDBFile if the error count is greater than $SOME_NUMBER. 
You could even make $SOME_NUMBER a parameter to readDBFile. 

Thanks, Bill Dudley 

-- 

the readDBFile procedure seems to be a custom procedure, how does it look like? where did you find it?

feel free to adjust it to your needs...

for example you could remove the double quote marks using the replace all block...

Taifun

-- 

readDBFile isn't custom, it's part of the standard "library", in "Procedures".

I certainly didn't write it.   I assume this is all in Java, of which I know nothing.

Yes, I could hack it to be less picky, assuming I wanted to learn Java, and figure out

how to add something to the library, but since it's part of the standard library, I thought

that the maintainers, who ever they are, could fix it.

Do the maintainers monitor this forum?  Is there a recommended way to submit bug reports?

Bill Dudley

-- 

readDBFile isn't custom, it's part of the standard "library", in "Procedures".

sorry, but the procedures drawer looks like this

if you upload a project from somewhere, then there might be additionally some custom procedures inside including their blocks

see also here how to work with procedures http://appinventor.mit.edu/explore/ai2/support/blocks/procedures.html ...

 Is there a recommended way to submit bug reports?

you can submit bug reports here in the forum, but in your case you have to fix your custom procedure yourself...

A very good way to learn App Inventor is to read the free Inventor's Manual here in the AI2 free online eBook http://www.appinventor.org/book2 ... the links are at the bottom of the Web page.  The book 'teaches' users how to program with AI2 blocks.

There is a free programming course here http://www.appinventor.org/content/CourseInABox/Intro  and the aia files for the projects in the book are here:  http://www.appinventor.org/bookFiles  

How to do a lot of basic things with App Inventor are described here:  http://www.appinventor.org/content/howDoYou/eventHandling .

Also do the tutorials http://appinventor.mit.edu/explore/ai2/tutorials.htmlto learn the basics of App Inventor, then try something and follow the

 Top 5 Tips: How to learn App Inventor

You will not find a tutorial, which does exactly what you are looking for. But doing the tutorials (not only reading a little bit) help you to understand, how things are working. This is important and this is the first step to do.

Taifun

-- 

Then I have no idea where readDBFile came from, and I'm not about to learn Java so I can fix someone else's

bad programming.  I'm a retired computer programmer, and I built my app to support another interest, so the

impetus isn't there for me to solve this.

One question: If I brought this additional code into my app inventor"library", is there a way I can see that

inclusion?  Is there something in the app inventor UI that will let me see other sources that might have been

dragged in?

If I can figure out who wrote this, at least I can submit the bug report to them.

Thanks for your patience.

Bill Dudley

-- 

It would really help if you provided a screenshot of your readDBFile procedure, so we can see what you are trying to do, and where the problem may be.

Taifun

-- 

I'm sorry, I've gotten terribly confused.  This is what I get for looking at a project a year after I wrote it.

On the other hand, that's when the bug report came in.

Oh my gosh, yes, DUH, I WROTE readDBFile.  However, I don't understand where the original error message is

coming from.  I know I didn't code that message.

mgnoc_contact.csv is a semicolon delimited text file that my app has downloaded from a web site,

When the app starts up, it does some init and then calls readDBFile

readDBFile looks like this:

and then, once dairectory is populated:

My code doesn't attempt to split any of the rows of the data file

into fields until the user selects one of the lines from a list.

I use semicolon as the field delimiter, so I'm not looking for

double quotes at all.  Yet double quotes in the csv file is

what are causing the error behaviour that caused me to start

this (admittedly embarrasing) forum post.

Thanks for your time and understanding.  All I can say in my defense

is that someday, you, too, will be old and easily confused as well.

Bill Dudley

-- 

in the procedure, does it make sense to assign an empty list to the listpicker?

the list from csv table block uses commas as delimiter, you said, your file uses semicolons as delimiter...

in your GotText event you can try to replace the list from csv tableblock by the split block, just split at \n (new line)...

Taifun

-- 

app

btw. why don't you use the Web component and its GotText event to retrieve the file into your app directly as text without need to first store it as file?

Taifun

-- 

Also consider replacing all semicolons with commas when the text arrives (a single block), then trying the csv table conversion block.

It might then be able to eat the double quotes while building the table.

ABG

-- 

 Taifun wrote:

Syntax Error: quote in unquoted cell" & "Cannot parse text argument to "list from csv table: as a CSV-formatted table"

However, I don't understand where the original error message is coming from.  I know I didn't code that message.

the error message is coming from the list from csv table block

there is something in your file, which is not as expected

Yes, why is "list from csv table" complaining about quotes in one of my fields?

in the procedure, does it make sense to assign an empty list to the listpicker?

I don't think this is what I'm doing: 

I initialize the global variable listLocations to an empty list.  Do I not have to initialize variables?

Then I call dairectory.ReadFrom the csv file.

That invokes my code to populate listLocations.

I then set ListPicker1 to the populated listLocations.

the list from csv table block uses commas as delimiter, you said, your file uses semicolons as delimiter...

in your GotText event you can try to replace the list from csv tableblock by the split block, just split at \n (new line)...

list from csv table is acting as a split on newline.  I tried to replace it with split on newline.  That turned out to not

work, apparently split on newline assumes CR LF, so it's deleting the first character of each line.  Not useful.

Here's what it looks like now:

 The first character of each line is missing.  This is not a usable result.  Any suggestions?

Taifun

-- 

Taifun wrote:

app

btw. why don't you use the Web component and its GotText event to download the file into your app directly as text without need to first store it as file?

I don't understand this, perhaps because I'm a unix programmer.  Where would the data reside, if not in a file?

How does it achieve persistence?  I don't want to have to download it every time the app runs.

Thanks,

Bill

Taifun

-- 

Some of my fields have commas in them.  I chose the semicolon on purpose to avoid collisions with likely data.

Bill

Abraham Getzler wrote:

Also consider replacing all semicolons with commas when the text arrives (a single block), then trying the csv table conversion block.

It might then be able to eat the double quotes while building the table.

ABG

-- 

Yes, why is "list from csv table" complaining about quotes in one of my fields?

without being able to see your file, unfortunately I don't know, 

you might want to provide an example file...

I don't think this is what I'm doing:
I initialize the global variable listLocations to an empty list.  Do I not have to initialize variables?

I was talking about this block

btw. why don't you use the Web component and its GotText event to download the file into your app directly as text without need to first store it as file?
I don't understand this, perhaps because I'm a unix programmer.  Where would the data reside, if not in a file?
How does it achieve persistence?  I don't want to have to download it every time the app runs.

usually you would download the file as text, convert it into a list and store it in TinyDB

an example of your file would help to give better suggestions.. .

Taifun

-- 

Taifun wrote:

Yes, why is "list from csv table" complaining about quotes in one of my fields?

without being able to see your file, unfortunately I don't know, 

you might want to provide an example file...

head -3 mgnoc_contact.csv

AB;Calgary;Greg;403 850 9979;;AC C DA HS KA LC LP T TR WC WD WE

AB;Calgary;Malcolm;403 862 0454;;AC C HS KA LC MO PT SP T WC

AB;Edmonton;;800 438 3246;780 447 3246;Dealer

etc. etc for 750 lines.

I don't think this is what I'm doing:
I initialize the global variable listLocations to an empty list.  Do I not have to initialize variables?

I was talking about this block

I do not understand your criticism of this block.

btw. why don't you use the Web component and its GotText event to download the file into your app directly as text without need to first store it as file?
I don't understand this, perhaps because I'm a unix programmer.  Where would the data reside, if not in a file?
How does it achieve persistence?  I don't want to have to download it every time the app runs.

usually you would download the file as text, convert it into a list and store it in TinyDB

What is the benefit of the additional conversion from text file to TinyDB?

an example of your file would help to give better suggestions.. .

Taifun

-- 

I re-tried again to use split on newline, and now it's working.  I swear I have no idea

why I was losing the first character of each line before.

Here's the cleaned up code:

This would seem to solve my immediate problem of list from csv table choking if there's a double quote in the file.

Thanks for your patience and your help.  I'm interested in learning more if you wish to continue the conversation.

Bill Dudley

Taifun wrote:

Syntax Error: quote in unquoted cell" & "Cannot parse text argument to "list from csv table: as a CSV-formatted table"

However, I don't understand where the original error message is coming from.  I know I didn't code that message.

the error message is coming from the list from csv table block

there is something in your file, which is not as expected

in the procedure, does it make sense to assign an empty list to the listpicker?

the list from csv table block uses commas as delimiter, you said, your file uses semicolons as delimiter...

in your GotText event you can try to replace the list from csv tableblock by the split block, just split at \n (new line)...

Taifun

-- 

these 3 rows look fine, you have to look for a quote and you will find the row, which results in that issue

see again your error message Syntax Error: quote in unquoted cell

What is the benefit of the additional conversion from text file to TinyDB?

TinyDB is your data store, you said, you like to achieve persistence... 

see also the documentation http://ai2.appinventor.mit.edu/reference/components/storage.html#TinyDB

it probably helps to do some tutorials to learn the basics...

btw. your for each loop is not necessary, the split block already converts the file into a list, see also the documentation

you write

set global listLocations to split (get text, \n)

Taifun

-- 

Duh.  removed another superfluous variable.

Bill

-- 

 re-tried again to use split on newline, and now it's working.  I swear I have no idea

why I was losing the first character of each line before.

That is an unlikely error pattern.

I would love to see it at work.

The closest I could come to it would be thru some kind of error

at converting CRLF to LF, which usually happens at the Windows/unix file transfer boundary.

Even then, I would expect the error to happen at the end of a line, with the addition of an invisible CR.

ABG

-- 

try without loop like this

  set global listLocations to split (get text, \n)

Taifun

-- 

If your quotes are arriving sporadically, like only for fields containing commas,

you can easily filter them before the split at semicolon using a replace-all block

in your pipeline.

ABG

-- 

Taifun wrote:

these 3 rows look fine, you have to look for a quote and you will find the row, which results in that issue

see again your error message Syntax Error: quote in unquoted cell

I solved that problem many hours ago by doing a binary search for the offending line, and then fixed the

code that generates the file so it wouldn't put quotes in there anymore.  However, I wanted to solve the

general problem: how to keep the app from choking should a differentspurious character get in there?

That's how I knew it was a double quote that caused the problem: I found it the hard way.  That's why it

would be nice if list from csv table would at least give offending line number in the error message.

Of course, if I'd know what the error message meant "quote in unquoted cell" I might have figured this

out faster.  I'm not as swift as I used to be.

What is the benefit of the additional conversion from text file to TinyDB?

TinyDB is your data store, you said, you like to achieve persistence... 

see also the documentation http://ai2.appinventor.mit.edu/reference/components/storage.html#TinyDB

OK, TinyDB is same as a perl hash.  I understand that it give me persistence.  But I'm not sure how I would

us it in this instance, except to store the entire list of entries in there under a single key value.  Or perhaps store

all the entries for a single state under the key for that state (NJ -> New Jersey list, NY -> New York list, etc).

But that seems like more work for no benefit that I can see. 

it probably helps to do some tutorials to learn the basics...

Despite my rocky start in this thread, I'm actually an experience (retired) developer.  So I'm familiar with

various databases (sql, Berkeley, perl hash, etc). 

btw. your for each loop is not necessary, the split block already converts the file into a list, see also the documentation

just use these blocks instead

  set global listLocations to split (get text, \n)

Yes, oops, that's obvious now that you mention it.  Another vestigial bit left over from using "list from csv table".

 I'll try that and see what happens.

Taifun

-- 

Interestingly, that gives me an empty list as a result.  Or at least a list

that the subsequent code displays as an empty ListPicker1.

I have no idea why, it should work.

Bill

Taifun wrote:

try without loop like this

  set global listLocations to split (get text, \n)

Taifun

-- 

I agree that is unlikely.  I don't fully trust the connection between appinventor on my browser

and AI Companion.

Bill

Abraham Getzler wrote:

 re-tried again to use split on newline, and now it's working.  I swear I have no idea

why I was losing the first character of each line before.

That is an unlikely error pattern.

I would love to see it at work.

The closest I could come to it would be thru some kind of error

at converting CRLF to LF, which usually happens at the Windows/unix file transfer boundary.

Even then, I would expect the error to happen at the end of a line, with the addition of an invisible CR.

ABG

-- 

But I'm not sure how I would us it in this instance, except to store the entire list of entries in there under a single key value. 

yes exactly

there is no need to store a file and nobody would be able to delete that file by mistake

but you also could do it the old fashioned way as you currently do it

 I'm actually an experience (retired) developer.  So I'm familiar with various databases (sql, Berkeley, perl hash, etc). 

you might be an expert somewhere else, but you are an App Inventorbeginner (at least you ask beginner questions)
doing the tutorials help to learn the App Inventor basics

see also 

Top 5 Tips: How to learn App Inventor

Interestingly, that gives me an empty list as a result.  Or at least a list that the subsequent code displays as an empty ListPicker1.

my guess is, there is a timing issue in your blocks, 

It would really help if you provided a screenshot of your relevant blocks, especially where are you assigning the list to the listpicker, so we can see what you are trying to do, and where the problem may be.

Taifun

-- 

Side note on TinyDB - it can store and retrieve lists and lists of lists.

I mention it in case that facility is unavailable on the other platforms you mentioned.

ABG

-- 

My typing is horrendous.  Two typos in that last reply!

Anyway, I think I'll stick with "old fashioned" in this case.  The additional work to use TinyDB is not

worth the small improvement of making it difficult for the user to delete the db file, especially since

he can just download another one with the app.

Taifun wrote:

But I'm not sure how I would us it in this instance, except to store the entire list of entries in there under a single key value. 

yes exactly

there is no need to store a file and nobody would be able to delete that file by mistake

but you also could do it the old fashioned way as you currently do it

 I'm actually an experience (retired) developer.  So I'm familiar with various databases (sql, Berkeley, perl hash, etc). 

you might be an expert somewhere else, but you are an App Inventorbeginner (at least you ask beginner questions)
doing the tutorials help to learn the App Inventor basics

see also 

Top 5 Tips: How to learn App Inventor

Very true, I'm a beginner at this. And as a retiree, I get to pick my jobs, and this isn't one of them.

I've got the app running better, and "good enough", so I think I'll leave it as it is.

Interestingly, that gives me an empty list as a result.  Or at least a list that the subsequent code displays as an empty ListPicker1.

my guess is, there is a timing issue in your blocks, 

It would really help if you provided a screenshot of your relevantblocks, especially where are you assigning the list to the listpicker, so we can see what you are trying to do, and where the problem may be.

I think I've taken enough of your time.  I can't get too excited if the code isn't fully optimized; it works,

there are very few users, it's hobby stuff anyway; in short, it'll do as is.

Thank you VERY much for your time and patience.

If there's a next time (a bug I can't figure out) I'll try to not go off half cocked as I did this time.

Bill Dudley 

Taifun

--