Write CIFS file 'permission denied' (Linux)

[tai-yi]

Dear guys,

I got an error when I use my application write to a CIFS file folder mounted on host machine.

I tried use same user with shell script echo 123 > /mnt/debug/a.txt success.

After application error, there is a 0kb file in the folder, means the file is created but application cannot flush binary content into the file.

Mount options mount -t cifs --verbose -o domain=***,username=***,password='***',dir_mode=0777,file_mode=0777,noperm,rw \ //somehost/folder /mnt

error log

Unhandled exception. System.UnauthorizedAccessException: Access to the path '/mnt/debug/5d256383-48b6-49f3-99b7-d964c4a964f6' is denied.
 ---> System.IO.IOException: Permission denied
   --- End of inner exception stack trace ---
   at System.IO.FileStream.WriteNative(ReadOnlySpan`1 source)
   at System.IO.FileStream.FlushWriteBuffer()
   at System.IO.FileStream.FlushInternalBuffer()
   at System.IO.FileStream.Flush(Boolean flushToDisk)
   at System.IO.FileStream.Flush()
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.IO.StreamWriter.Dispose(Boolean disposing)
   at System.IO.TextWriter.Dispose()
   at System.IO.File.WriteAllText(String path, String contents)
   at NasDebugger.Program.Main(String[] args)
Aborted (core dumped)

sample code

static int Main(string[] args)
        {
            if (args.Length == 0 || string.IsNullOrEmpty(args[0]))
            {
                Console.WriteLine("please set dir");
                return -1;
            }
            var dir = args[0];
            if (!Directory.Exists(dir))
            {
                Console.WriteLine("please set right dir");
                return -1;
            }
            var id = Guid.NewGuid().ToString();
            File.WriteAllText(Path.Combine(dir, id), "123123");
            Console.WriteLine(id);
            return 0;
        }

Right now , I only can reproduce it in my datacenter😭.

OS Linux version 4.12.14-95.54-default (geeko@buildhost) (gcc version 4.8.5 (SUSE Linux) ) #1 SMP Thu Jun 4 12:49:28 UTC 2020 (892ef1f)
dotnetcore version: 3.1.8

Does anyone have idea about it?

[carlossanlop]

It seems you are hitting issue #1816, which has been fixed in 5.0 and master and was also backported to 3.1. The next servicing release of .NET Core 3.1 should contain the fix for this. It should go into the November servicing release. Or if you can, you could upgrade your application to 5.0 RC1 and the issue should be gone there.

[carlossanlop]

On a second thought, the exception is not exactly the same: Unhandled exception. System.UnauthorizedAccessException: Access to the path '/mnt/debug/5d256383-48b6-49f3-99b7-d964c4a964f6' is denied.

So let me ask you a couple of questions to make sure we are on the same page, @tai-yi :

• Do you have permission to write in the remote folder?
• Do you have enough disk space in the remote folder to save your file?
• Is your file larger than the available free disk space?

[wfurt]

best option how to troubleshoot this IMHO would be running the app under strace e.g. something like strace -r -f -s 100 -e trace=%file -e trace=%desc myapp Do not use dotnet run: either use dotnet myappl.dll or publish self-contained app.

[tai-yi]

@carlossanlop

• Do you have permission to write in the remote folder?

    yes, I can write with shell script 'echo 123 > /mnt/debug/test'
  

• Do you have enough disk space in the remote folder to save your file?

    yes, I can write successfuly with shell script
  

• Is your file larger than the available free disk space?

    no, I just write text `123123` to the file
  

[tai-yi]

Update

thanks @wfurt, I will try as your suggestion, this result tested by single file application.

I tried use 4 ways to write file.

1. File.WriteAllText
2. new StreamWriter(path)
3. new FileStream(path, FileMode.OpenOrCreate, FileAccess.Write, FileShare.None)
4. new FileStream(path, FileMode.OpenOrCreate, FileAccess.Write, FileShare.Read)

Only 3 works fine,, so FileShare should impact the result.

Sample code

class Program
{
    static int Main(string[] args)
    {
        if (args.Length == 0 || string.IsNullOrEmpty(args[0]))
        {
            Console.WriteLine("please set dir");
            return -1;
        }
        var dir = args[0];

        if (!Directory.Exists(dir))
        {
            Console.WriteLine("please set and existing dir");
            return -1;
        }
        var id = Guid.NewGuid().ToString();
        var model = 1;
        if (args.Length > 1) {
            if (!int.TryParse(args[1], out model))
                model = 1;
        }
        var path = Path.Combine(dir, id);
        
        switch (model)
        {
            case 1:
                Write1(path);break;
            case 2:
                Write2(path); break;
            case 3:
                Write3(path); break;
            case 4:
                Write4(path); break;
            default:
                return -1;
        }           
        return 0;
    }

    static void Write1(string path)
    {
        File.WriteAllText(path, "123123");
        Console.WriteLine(path);
    }

    static void Write2(string path)
    {
        using var writer = new StreamWriter(path);
        writer.Write("123123");
        Console.WriteLine(path);
    }

    static void Write3(string path)
    {
        using var fs = new FileStream(path, FileMode.OpenOrCreate, FileAccess.Write, FileShare.None);
        using var writer = new StreamWriter(fs);
        writer.Write("123123");
        Console.WriteLine(path);
    }

    static void Write4(string path)
    {
        using var fs = new FileStream(path, FileMode.OpenOrCreate, FileAccess.Write, FileShare.Read);
        using var writer = new StreamWriter(fs);
        writer.Write("123123");
        Console.WriteLine(path);
    }
}

sample log

the-host-name:/home/debugger # ./NasDebugger /mnt/debug/ 1
Unhandled exception. System.UnauthorizedAccessException: Access to the path '/mnt/debug/c5d0bccf-edc8-4b7b-8607-fab879d0da79' is denied.
 ---> System.IO.IOException: Permission denied
   --- End of inner exception stack trace ---
   at System.IO.FileStream.WriteNative(ReadOnlySpan`1 source)
   at System.IO.FileStream.FlushWriteBuffer()
   at System.IO.FileStream.FlushInternalBuffer()
   at System.IO.FileStream.Flush(Boolean flushToDisk)
   at System.IO.FileStream.Flush()
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.IO.StreamWriter.Dispose(Boolean disposing)
   at System.IO.TextWriter.Dispose()
   at System.IO.File.WriteAllText(String path, String contents)
   at NasDebugger.Program.Write1(String path)
   at NasDebugger.Program.Main(String[] args)
Aborted (core dumped)
the-host-name:/home/debugger # ./NasDebugger /mnt/debug/ 2
/mnt/debug/5a973cdd-ce73-40ad-b483-507f3843cbce
Unhandled exception. System.UnauthorizedAccessException: Access to the path '/mnt/debug/5a973cdd-ce73-40ad-b483-507f3843cbce' is denied.
 ---> System.IO.IOException: Permission denied
   --- End of inner exception stack trace ---
   at System.IO.FileStream.WriteNative(ReadOnlySpan`1 source)
   at System.IO.FileStream.FlushWriteBuffer()
   at System.IO.FileStream.FlushInternalBuffer()
   at System.IO.FileStream.Flush(Boolean flushToDisk)
   at System.IO.FileStream.Flush()
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.IO.StreamWriter.Dispose(Boolean disposing)
   at System.IO.TextWriter.Dispose()
   at NasDebugger.Program.Write2(String path)
   at NasDebugger.Program.Main(String[] args)
Aborted (core dumped)
the-host-name:/home/debugger # ./NasDebugger /mnt/debug/ 3
/mnt/debug/2e23f4c7-ae3d-488d-968a-a98830777305
the-host-name:/home/debugger # ./NasDebugger /mnt/debug/ 4
/mnt/debug/d82757c5-9b77-4ee0-93c5-c140a4c39f8f
Unhandled exception. System.UnauthorizedAccessException: Access to the path '/mnt/debug/d82757c5-9b77-4ee0-93c5-c140a4c39f8f' is denied.
 ---> System.IO.IOException: Permission denied
   --- End of inner exception stack trace ---
   at System.IO.FileStream.WriteNative(ReadOnlySpan`1 source)
   at System.IO.FileStream.FlushWriteBuffer()
   at System.IO.FileStream.FlushInternalBuffer()
   at System.IO.FileStream.Flush(Boolean flushToDisk)
   at System.IO.FileStream.Flush()
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.IO.StreamWriter.Dispose(Boolean disposing)
   at System.IO.TextWriter.Dispose()
   at NasDebugger.Program.Write4(String path)
   at NasDebugger.Program.Main(String[] args)
Aborted (core dumped)

[wfurt]

the point is not to have single application @tai-yi but to get insight to low-level kernel calls. Pick one of the failing scenarios and get the trace (you can also use -o trace.txt)

[tai-yi]

@wfurt , thanks for your help.

I tried run command strace -r -f -s 100 -e trace=file -P /mnt/debug/ -o trace2 ./NasDebugger /mnt/debug/ 2

here is the content in trace2

31866      0.000000 stat("/mnt/debug/", {st_mode=S_IFDIR|0777, st_size=0, ...}) = 0
31866      0.084994 --- SIGABRT {si_signo=SIGABRT, si_code=SI_TKILL, si_pid=31866, si_uid=0} ---
31871      0.003428 +++ killed by SIGABRT (core dumped) +++
31873      0.000018 +++ killed by SIGABRT (core dumped) +++
31869      0.000007 +++ killed by SIGABRT (core dumped) +++
31868      0.000006 +++ killed by SIGABRT (core dumped) +++
31867      0.000006 +++ killed by SIGABRT (core dumped) +++
31870      0.003771 +++ killed by SIGABRT (core dumped) +++
31872      0.000010 +++ killed by SIGABRT (core dumped) +++
31874      0.004476 +++ killed by SIGABRT (core dumped) +++
31866      0.000066 +++ killed by SIGABRT (core dumped) +++

[wfurt]

There is not much. I think you will also need the -e trace=%desc. And I assume the SIGABRT is from unhandled execution. (it may be better to catch it and print error message to be sure)

[tai-yi]

@wfurt , new trace file uploaded.
trace2.txt

[wfurt]

The interesting entry is:

29766      0.003445 open("/mnt/debug/eda7ab5a-1a3c-4921-a3b8-b33afba840e9", O_WRONLY|O_CREAT|O_CLOEXEC, 0666) = 23

so the open call fails with ENFILE errno. You can try to check with lsof to see what files are opened.
https://stackoverflow.com/questions/24862733/difference-between-linux-errno-23-and-linux-errno-24 may also have some hints.

One can ask why echo works: as you may see from the trace, dotnet opens lot of file to run. So if you are close to limit that may be just enough. My recommendation would be to check ulimit and fs.file-max and try it again.

I think we can probably also do better job mapping that particular error. (assuming that the run producing the trace still failed with "Access denied")