Weird library compile message

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Weird library compile message

jamshark70-2
ERROR: Could not iterate on '/home/dlm/.local/share/SuperCollider/downloaded-quarks/ddwChucklib-livecode/.#cl-manual.org': No such file or directory

What. Is. That. ?

It's not a .sc file. It shouldn't affect library compilation.

The result is that library compilation ignores the whole directory... if I have a file in that directory open in Emacs, with unsaved changes.

Yeah, that makes perfect sense...

hjh


_______________________________________________
sc-dev mailing list

info (subscription, etc.): http://www.birmingham.ac.uk/facilities/ea-studios/research/supercollider/mailinglist.aspx
archive: http://www.listarc.bham.ac.uk/marchives/sc-dev/
search: http://www.listarc.bham.ac.uk/lists/sc-dev/search/
Reply | Threaded
Open this post in threaded view
|

Re: Weird library compile message

brianlheim
Hey James,

I wrote the relevant code. I'm not sure what's happening there, but will look into it. It looks like several edge cases (dotfile, temp file, uncommon character in filename), should be interesting to pin this down.

Would you like the error message to say something else? I guess 'iterate' is kind of an implementation detail.

Brian


On Jan 11, 2018 5:08 AM, <[hidden email]> wrote:
ERROR: Could not iterate on '/home/dlm/.local/share/SuperCollider/downloaded-quarks/ddwChucklib-livecode/.#cl-manual.org': No such file or directory

What. Is. That. ?

It's not a .sc file. It shouldn't affect library compilation.

The result is that library compilation ignores the whole directory... if I have a file in that directory open in Emacs, with unsaved changes.

Yeah, that makes perfect sense...

hjh


_______________________________________________
sc-dev mailing list

info (subscription, etc.): http://www.birmingham.ac.uk/facilities/ea-studios/research/supercollider/mailinglist.aspx
archive: http://www.listarc.bham.ac.uk/marchives/sc-dev/
search: http://www.listarc.bham.ac.uk/lists/sc-dev/search/

Reply | Threaded
Open this post in threaded view
|

Re: Weird library compile message

jamshark70-2

On January 11, 2018 22:22:36 [hidden email] wrote:

> I wrote the relevant code. I'm not sure what's happening there, but will
> look into it. It looks like several edge cases (dotfile, temp file,
> uncommon character in filename), should be interesting to pin this down.
>
> Would you like the error message to say something else? I guess 'iterate'
> is kind of an implementation detail.


I think library compilation should disregard files whose extension is not ".sc" regardless of any unusual characters preceding it.

I wonder if it's reading the filename as a directory name (like . or ..) but failing to access the non-directory's contents.

Sorry for temper. I'm nearing release of some code, and I was in a creative flow and suddenly my whole live coding framework couldn't load... and I also found out today that my school is taking away my whole weekend too. So I was under some time pressure. Not a good time for mysterious behavior. (At least there *is* a message -- silent failure would have been impossible to figure out.)

It's ok, I know how to prevent it now.

hjh

Sent with AquaMail for Android
http://www.aqua-mail.com

Reply | Threaded
Open this post in threaded view
|

Re: Weird library compile message

brianlheim
I wonder if it's reading the filename as a directory name (like . or ..) but failing to access the non-directory's contents

The code uses boost filesystem's recursive directory iterator, whose iterate method tries to move to the next entry in the directory, and may also try to push into directory objects or pop out of empty directories. So it might indeed think that's a directory. Probably a bug in my code, but it could be a subtlety in the filesystem or a bug in bfs. Probably my code though haha.

On Jan 11, 2018 9:34 AM, <[hidden email]> wrote:

On January 11, 2018 22:22:36 [hidden email] wrote:

> I wrote the relevant code. I'm not sure what's happening there, but will
> look into it. It looks like several edge cases (dotfile, temp file,
> uncommon character in filename), should be interesting to pin this down.
>
> Would you like the error message to say something else? I guess 'iterate'
> is kind of an implementation detail.


I think library compilation should disregard files whose extension is not ".sc" regardless of any unusual characters preceding it.

I wonder if it's reading the filename as a directory name (like . or ..) but failing to access the non-directory's contents.

Sorry for temper. I'm nearing release of some code, and I was in a creative flow and suddenly my whole live coding framework couldn't load... and I also found out today that my school is taking away my whole weekend too. So I was under some time pressure. Not a good time for mysterious behavior. (At least there *is* a message -- silent failure would have been impossible to figure out.)

It's ok, I know how to prevent it now.

hjh

Sent with AquaMail for Android
http://www.aqua-mail.com

Reply | Threaded
Open this post in threaded view
|

Re: Weird library compile message

brianlheim
I found I am able to reproduce this locally. It looks like the .#file.org file is a symlink pointing to a non-existent file. This appears to be normal behavior for emacs, although it can be changed.


The algorithm for finding files to compile in sclang is essentially:

if (at a directory)
  if (should skip the directory [dotfile, already been compiled, etc.])
    move to next entry
resolve an alias (on macOS only)
if (was an alias and alias pointed to a directory)
  try to compile that directory [non-aliased directories will be walked as usual]
else
  try to compile the current file [with separate logic]
move to next entry

I used an option for recursive_directory_iterator that follows all symlinks, and intentionally made it fail when failing to iterate on a broken symlink, because this seemed like the simplest way to match behavior before the rewrite.

Now, I'm thinking the best thing to do might be:
- don't automatically recurse on symlinks
- skip any dotfile immediately
- try to follow symlinks, and warn (not fail) on a broken symlink

That would get rid of the error and warning in your case, but still retain most of the old behavior. The _one case_ where this would make a difference is if you had a symlink ".dir" pointing to some "dir". Those would be compiled under the current method, but not with the changes I gave above. Just tested this locally to make sure.

Here's a full enumeration of current behavior. The things that would change are bolded. I use "dot" and "normal" for names that start with a dot vs. names that don't.

normal .sc file: compiled
normal dir: compiled
dot file or dir: not compiled
dot symlink -> normal .sc file: not compiled
dot symlink -> normal dir: compiled (becomes not compiled)
dot symlink -> dot anything: not compiled
normal symlink -> normal anything: compiled
normal symlink -> dot anything: not compiled
broken symlink: error on current compilation directory (becomes warning, and compilation continues)

-Brian

On Thu, Jan 11, 2018 at 9:39 AM, brian heim <[hidden email]> wrote:
I wonder if it's reading the filename as a directory name (like . or ..) but failing to access the non-directory's contents

The code uses boost filesystem's recursive directory iterator, whose iterate method tries to move to the next entry in the directory, and may also try to push into directory objects or pop out of empty directories. So it might indeed think that's a directory. Probably a bug in my code, but it could be a subtlety in the filesystem or a bug in bfs. Probably my code though haha.

On Jan 11, 2018 9:34 AM, <[hidden email]> wrote:

On January 11, 2018 22:22:36 [hidden email] wrote:

> I wrote the relevant code. I'm not sure what's happening there, but will
> look into it. It looks like several edge cases (dotfile, temp file,
> uncommon character in filename), should be interesting to pin this down.
>
> Would you like the error message to say something else? I guess 'iterate'
> is kind of an implementation detail.


I think library compilation should disregard files whose extension is not ".sc" regardless of any unusual characters preceding it.

I wonder if it's reading the filename as a directory name (like . or ..) but failing to access the non-directory's contents.

Sorry for temper. I'm nearing release of some code, and I was in a creative flow and suddenly my whole live coding framework couldn't load... and I also found out today that my school is taking away my whole weekend too. So I was under some time pressure. Not a good time for mysterious behavior. (At least there *is* a message -- silent failure would have been impossible to figure out.)

It's ok, I know how to prevent it now.

hjh

Sent with AquaMail for Android
http://www.aqua-mail.com


Reply | Threaded
Open this post in threaded view
|

Re: Weird library compile message

jamshark70-2
---- On Fri, 12 Jan 2018 01:00:00 +0800 <[hidden email]> wrote ----
> I found I am able to reproduce this locally. It looks like the .#file.org file is a symlink pointing to a non-existent file. This appears to be normal behavior for emacs, although it can be changed.
>
> I used an option for recursive_directory_iterator that follows all symlinks, and intentionally made it fail when failing to iterate on a broken symlink, because this seemed like the simplest way to match behavior before the rewrite.
> Now, I'm thinking the best thing to do might be:- don't automatically recurse on symlinks- skip any dotfile immediately
> - try to follow symlinks, and warn (not fail) on a broken symlink
> That would get rid of the error and warning in your case, but still retain most of the old behavior. The _one case_ where this would make a difference is if you had a symlink ".dir" pointing to some "dir". Those would be compiled under the current method, but not with the changes I gave above. Just tested this locally to make sure.

I think I'd be OK with that. Certainly it would be good to have a warning for symlinks to nowhere.

I had been thinking we should see that the extension is not '.sc' and ignore it without an error, but if it's a broken symlink, there's no way to determine whether the target was supposed to be a file or directory, so I guess the warning really is necessary.

I don't remember the old behavior (and I don't remember seeing this problem before), but... what I observed yesterday is that the broken symlink failed not only the symlink, but all of the symlink's siblings in the same parent directory. I had ./.#cl-manual.org (the broken link) and ./clLiveCode-ext.sc, and the broken link caused the '.sc' file to be skipped as well. Did it really do that before? Even if so, I think that behavior should change.

It may not be necessary to skip all dotfiles. Perhaps, if the broken link is a dotfile, suppress the warning.

Thanks for the quick look at it. Not urgent -- it can definitely wait for 3.9.1.

hjh


_______________________________________________
sc-dev mailing list

info (subscription, etc.): http://www.birmingham.ac.uk/facilities/ea-studios/research/supercollider/mailinglist.aspx
archive: http://www.listarc.bham.ac.uk/marchives/sc-dev/
search: http://www.listarc.bham.ac.uk/lists/sc-dev/search/
Reply | Threaded
Open this post in threaded view
|

Re: Weird library compile message

brianlheim
> Did it really do that before?

I'm really not sure, sorry. Probably not.

> Thanks for the quick look at it. Not urgent -- it can definitely wait for 3.9.1.

No problem. For now a workaround would be to config emacs so it doesn't create broken symlinks, or so that it creates temp files in a separate directory. :)

-Brian

On Thu, Jan 11, 2018 at 7:55 PM, <[hidden email]> wrote:
---- On Fri, 12 Jan 2018 01:00:00 +0800 <[hidden email]> wrote ----
> I found I am able to reproduce this locally. It looks like the .#file.org file is a symlink pointing to a non-existent file. This appears to be normal behavior for emacs, although it can be changed.
>
> I used an option for recursive_directory_iterator that follows all symlinks, and intentionally made it fail when failing to iterate on a broken symlink, because this seemed like the simplest way to match behavior before the rewrite.
> Now, I'm thinking the best thing to do might be:- don't automatically recurse on symlinks- skip any dotfile immediately
> - try to follow symlinks, and warn (not fail) on a broken symlink
> That would get rid of the error and warning in your case, but still retain most of the old behavior. The _one case_ where this would make a difference is if you had a symlink ".dir" pointing to some "dir". Those would be compiled under the current method, but not with the changes I gave above. Just tested this locally to make sure.

I think I'd be OK with that. Certainly it would be good to have a warning for symlinks to nowhere.

I had been thinking we should see that the extension is not '.sc' and ignore it without an error, but if it's a broken symlink, there's no way to determine whether the target was supposed to be a file or directory, so I guess the warning really is necessary.

I don't remember the old behavior (and I don't remember seeing this problem before), but... what I observed yesterday is that the broken symlink failed not only the symlink, but all of the symlink's siblings in the same parent directory. I had ./.#cl-manual.org (the broken link) and ./clLiveCode-ext.sc, and the broken link caused the '.sc' file to be skipped as well. Did it really do that before? Even if so, I think that behavior should change.

It may not be necessary to skip all dotfiles. Perhaps, if the broken link is a dotfile, suppress the warning.

Thanks for the quick look at it. Not urgent -- it can definitely wait for 3.9.1.